To use s3backer, you need a filesystem to play with. This page describes how to create one.
For an example of an existing filesystem, see RunningTheDemo.
To create an encrypted filesystem, read this page first and then see CreatingAnEncryptedFilesystem.
Creating your own new filesystem is very easy. Just follow these simple steps:
- Download, build, and install s3backer.
- Put your Amazon S3 access ID and access key (as a colon-separated pair) in ~/.s3backer_passwd like this (the information below is an example only):
# my S3 account accessID and accessKey
0KAODKRXJM39543K343:+MkIE9MA/dkwEaldRoaPP83dfa03=
- Decide how big you want your filesystem to be. Let's say we want a modest 1 terabyte of storage.
- Decide what you want your s3backer block size to be. See ChoosingBlockSize for more info. You may want to use a larger block size than the default of 4k. Let's use 128k for this example.
- Decide what bucket you want to store the filesystem blocks in. For this example, we'll use mybucket. This is just an example; you will have to create and use your own bucket of course.
- Create a directory where the s3backer filesystem can be mounted: mkdir ~/mnt-s3b
- Fire up s3backer: s3backer --blockSize=128k --size=1t --listBlocks mybucket ~/mnt-s3b.
- Now create your filesystem, pretending that ~/mnt-s3b/file was a block device such as a disk partition: mkreiserfs -f -b 4096 -s 513 ~/mnt-s3b/file. Of course, you don't have to use the Reiser filesystem, you can use ext2, XFS, minix, NTFS, or whatever.
- Finally, mount your newly-created filesystem using a loopback mount: sudo mount -o loop ~/mnt-s3b/file /mnt
- You now have 1 terabyte of storage mounted under /mnt to play with!
Note: the number of blocks that have to be written to initialize a filesystem (and therefore the amount of time it takes) depends entirely on the filesystem. Use --listBlocks to ensure that if any of those blocks are all zeroes, no network traffic has to occur.
Experience has shown that the ext2 filesystem writes a lots and lots of zero blocks when initializing large filesystems. You might consider reiserfs or xfs instead.
When initializing Reiserfs, you can look for the "Number of blocks consumed by mkreiserfs formatting process: NNNN" message.
Examples of filesystem initialization commands:
- Reiser: mkreiserfs -f -b 4096 -s 513 ~/mnt-s3b/file
- ext4: mke2fs -t ext4 -E nodiscard -F ~/mnt-s3b/file
- XFS: mkfs.xfs ~/mnt-s3b/file
To unmount your filesystem:
- Unmount the "upper" filesystem: sudo umount /mnt
- Unmount the s3backer filesystem: sudo umount ~/mnt-s3b
and just to be crystal clear -- this does not create on s3 16 terabytes of data. Amazon only sees what is actually used on the file system and so will charge accordingly...
Thanks
what if the --size=... is incorrectly set, and one wishes to "delete" the ~/mnt-s3b/file and ~/mnt-s3b/stats? rm doesn't work here. Any tips are appreciated.
If you don't care about deleting the old data, just unmount it then remount it with the --force flag.
To completely erase the filesystem, unmount it first, then remove the blocks from Amazon S3. You can do this in a few ways:
just for more clarity the name 'bigspace' is not some type of bucket that you have to pick from a limited number of buckets. It is the name of the bucket that you set. This one had be going for awhile. :-)
Just wanted to put out a word of warning. s3backer is great for it's designed uses, as a filesystem in which you care about random reads/writes at the block level, but CAN BE VERY EXPENSIVE to use as a backup method. For example, I backed up about 50 gigs of data, and since I used the default of a 4K block size, there was an exorbitant number of PUT requests (uploads) for all of the data resulting in a bill > $90 just for PUT requests. So be careful with how you use it.
I was able to create a new filesystem, BUT:
Nothing happens. If i put files in my mounted filesystem, the files don't get transferred to AWS. I was running it for some time and I realized that there are MANY requests to AWS. In the AWS Account Activity, there are about 7.000 requests and counting. Is there anything I forgot?
herr.schnegg: join the discussion group and we can try to answer your question.
Also (unrelated): regarding my comment on Sep 01, 2008: newer versions of s3backer now support the --erase flag. See man page for details.
when i reboot the PC,the filesystem that i create for test is unmounted,and how can i get the file stored in the old filesystem,in other words,how can i get the file which is stored in last time.
Just run s3backer again with the same configuration to remount.
yes，i have a try to remount.Before remount,there are some blocks in my test directory “mybucket“，and i run s3backer again with the same configuration to remount，after that ,i can't find the file which is writed last time in the /mnt.
For help with problems you can email the s3backer-devel group:
http://groups.google.com/group/s3backer-devel