quillen - Project Hosting on Google Code

Last 30 days Nov 30, 2009 issue 8 (org.apache.commons.httpclient.HttpMethodDirector executeWith...) reported by xpd259 - What steps will reproduce the problem? 1. java -jar quillen-0.4.jar -command backup -base ~/Pictures -snapshot pics-`date +%Y` 2> quillen.log 2. files upload 3. What is the expected output? What do you see instead? just a list of files uploaded INFO: backed up file Photos/2009/09/20/dscf2254.jpg INFO: backed up file Photos/2009/10/22/DSCF2420.JPG INFO: backed up file 2009/04/20/DSCF1142.JPG INFO: backed up file Photos/2009/09/20/dscf2236.jpg INFO: backed up file 2008/11/08/dscf0978.jpg 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server sdb.amazonaws.com failed to respond 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request 30-Nov-2009 21:20:06 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:20:07 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 204519 bytes 30-Nov-2009 21:21:05 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:21:06 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 49146 bytes What version of the product are you using? On what operating system? 0.4 ubuntu linux 9.10 Please provide any additional information below. I am trying to upload all my ~/Pictures and it is =>16GB it's hard to tell with other applications using the net but my upload data seems to carry on at 181kb/s even after this error but no matter how many hours I leave it I never get more listings of files been uploaded What steps will reproduce the problem? 1. java -jar quillen-0.4.jar -command backup -base ~/Pictures -snapshot pics-`date +%Y` 2> quillen.log 2. files upload 3. What is the expected output? What do you see instead? just a list of files uploaded INFO: backed up file Photos/2009/09/20/dscf2254.jpg INFO: backed up file Photos/2009/10/22/DSCF2420.JPG INFO: backed up file 2009/04/20/DSCF1142.JPG INFO: backed up file Photos/2009/09/20/dscf2236.jpg INFO: backed up file 2008/11/08/dscf0978.jpg 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server sdb.amazonaws.com failed to respond 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request 30-Nov-2009 21:20:06 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:20:07 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 204519 bytes 30-Nov-2009 21:21:05 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:21:06 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 49146 bytes What version of the product are you using? On what operating system? 0.4 ubuntu linux 9.10 Please provide any additional information below. I am trying to upload all my ~/Pictures and it is =>16GB it's hard to tell with other applications using the net but my upload data seems to carry on at 181kb/s even after this error but no matter how many hours I leave it I never get more listings of files been uploaded Earlier this year Nov 28, 2009 issue 7 (NFO: nothing to backup) commented on by xpd259 - ah spotted the error i needed to sudo 1st as /home/xpd259 was own by root ah spotted the error i needed to sudo 1st as /home/xpd259 was own by root Nov 28, 2009 issue 7 (NFO: nothing to backup) reported by xpd259 - What steps will reproduce the problem? 1. find /home/xpd259 \| java -jar quillen-0.4.jar -command backup 2> quillen.log 2.cat guillen.log = INFO: nothing to backup 3. What is the expected output? What do you see instead? not sure as it's my 1st run but i expect some form of files been transfered What version of the product are you using? On what operating system? 0.4 Please provide any additional information below. running ubuntu 9.10 i686 What steps will reproduce the problem? 1. find /home/xpd259 \| java -jar quillen-0.4.jar -command backup 2> quillen.log 2.cat guillen.log = INFO: nothing to backup 3. What is the expected output? What do you see instead? not sure as it's my 1st run but i expect some form of files been transfered What version of the product are you using? On what operating system? 0.4 Please provide any additional information below. running ubuntu 9.10 i686 Sep 30, 2009 issue 6 (Qullien does not resume interrupted transfers.) commented on by linuxfood - Ahh, I see. Thanks for the rundown. I should have looked at the code before making an assumption. Instead of every file/chunk, capturing interrupts (and unclean exists where possible) and storing size metadata would be enough, I think. Possibly some small local metadata could be used to update snapshot sizes after the fact? I'm still interested in implementing when we reach a solution. Even a small UI improvement in this area could help, I think. Side note, this should probably be changed to an "enhancement" instead of a defect, but google code doesn't appear to allow me to do that. Thanks for making quillen! Ahh, I see. Thanks for the rundown. I should have looked at the code before making an assumption. Instead of every file/chunk, capturing interrupts (and unclean exists where possible) and storing size metadata would be enough, I think. Possibly some small local metadata could be used to update snapshot sizes after the fact? I'm still interested in implementing when we reach a solution. Even a small UI improvement in this area could help, I think. Side note, this should probably be changed to an "enhancement" instead of a defect, but google code doesn't appear to allow me to do that. Thanks for making quillen! Aug 27, 2009 issue 6 (Qullien does not resume interrupted transfers.) commented on by grourk - Yeah, since large files are split into chunks (averaging 128KB) -- and since the chunks are uploaded individually -- those that were uploaded successfully will not get re-uploaded. Metadata about the snapshot isn't updated after every file gets uploaded, only once at the beginning (with the 0 size you see) and once at the end. It could be modified to update that size after every file and/or chunk is completed. However, in many cases (like with lots of small files) that would be a performance overhead. Perhaps this could be an option you can toggle with a config file or a command line flag? It's great to hear feedback about how you use Quillen and how it could be improved! Thanks! Yeah, since large files are split into chunks (averaging 128KB) -- and since the chunks are uploaded individually -- those that were uploaded successfully will not get re-uploaded. Metadata about the snapshot isn't updated after every file gets uploaded, only once at the beginning (with the 0 size you see) and once at the end. It could be modified to update that size after every file and/or chunk is completed. However, in many cases (like with lots of small files) that would be a performance overhead. Perhaps this could be an option you can toggle with a config file or a command line flag? It's great to hear feedback about how you use Quillen and how it could be improved! Thanks! Aug 27, 2009 issue 6 (Qullien does not resume interrupted transfers.) commented on by linuxfood - Awesome. Does the "resume" occur even on partial transfers of files? I have some very large >100GB files, that I'd really rather not pay to upload a second time in the event my internet goes out. If it does, I would be happy to improve the UI around this area, because from my testing, I got a snapshot of 0 size, instead of the actual transferred size. Awesome. Does the "resume" occur even on partial transfers of files? I have some very large >100GB files, that I'd really rather not pay to upload a second time in the event my internet goes out. If it does, I would be happy to improve the UI around this area, because from my testing, I got a snapshot of 0 size, instead of the actual transferred size. Aug 27, 2009 issue 6 (Qullien does not resume interrupted transfers.) commented on by grourk - Thanks for volunteering to contribute! As far as partial snapshotting goes, this already happens due to the nature of how Quillen works. It uploads each file individually (and splits large files into chunks). If it has uploaded, say, 10 out of 100 files in a snapshot when it dies (or loses connectivity), then when it starts up again, it will see that 10 files haven't changed and will continue on the rest. This is a big gain if those ten files are large. This is the same mechanism that prevents it from re-uploading data that hasn't changed from snapshot to snapshot. For instance, if a file hasn't changed at all, or only one of its chunks has changed. There is hashing of those 10 files that obviously has a nonzero cost, and this could be optimized. What I have in mind is to look at file modification times and skip hashing unmodified files. This would likely involve a local cache of the metadata stored in SimpleDB. If you have more ideas there, let me know! Thanks for volunteering to contribute! As far as partial snapshotting goes, this already happens due to the nature of how Quillen works. It uploads each file individually (and splits large files into chunks). If it has uploaded, say, 10 out of 100 files in a snapshot when it dies (or loses connectivity), then when it starts up again, it will see that 10 files haven't changed and will continue on the rest. This is a big gain if those ten files are large. This is the same mechanism that prevents it from re-uploading data that hasn't changed from snapshot to snapshot. For instance, if a file hasn't changed at all, or only one of its chunks has changed. There is hashing of those 10 files that obviously has a nonzero cost, and this could be optimized. What I have in mind is to look at file modification times and skip hashing unmodified files. This would likely involve a local cache of the metadata stored in SimpleDB. If you have more ideas there, let me know! Aug 25, 2009 issue 6 (Qullien does not resume interrupted transfers.) reported by linuxfood - I am interested in implementing this feature, and would like to collaborate so as to reduce duplicated effort. What steps will reproduce the problem? 1. Start up a backup with Qullien 2. Press control-C, unplug your network cable, or otherwise break the transfer. 3. Start the same backup with the same data What is the expected output? What do you see instead? Expected: Have quillen note that it was interrupted during the last transfer, and resume. Actual: A zero size snapshot is left over, and all data must be re-transferred. There's a couple different ways that quillen could handle this situation, I think. 1) Quillen could support a "partial snapshot" which should be linked to it's children. This option seems the least favorable as it would hinder the "atomicity" of a snapshot. 2) Quillen can calculate the delta between the partial, failed, snapshot and then merge the failed snapshot with a new one, thus only transferring the delta. I need to think a little bit more on it, but I'm reasonably sure that option #2 will work even in the event of the data changing between the time that the first transfer failed, and the second transfer. I am interested in implementing this feature, and would like to collaborate so as to reduce duplicated effort. What steps will reproduce the problem? 1. Start up a backup with Qullien 2. Press control-C, unplug your network cable, or otherwise break the transfer. 3. Start the same backup with the same data What is the expected output? What do you see instead? Expected: Have quillen note that it was interrupted during the last transfer, and resume. Actual: A zero size snapshot is left over, and all data must be re-transferred. There's a couple different ways that quillen could handle this situation, I think. 1) Quillen could support a "partial snapshot" which should be linked to it's children. This option seems the least favorable as it would hinder the "atomicity" of a snapshot. 2) Quillen can calculate the delta between the partial, failed, snapshot and then merge the failed snapshot with a new one, thus only transferring the delta. I need to think a little bit more on it, but I'm reasonably sure that option #2 will work even in the event of the data changing between the time that the first transfer failed, and the second transfer. Aug 09, 2009 issue 5 (Uppercase bucket names) Status changed by grourk - Fixed in v0.4 Status: Fixed Fixed in v0.4 Status: Fixed Aug 09, 2009 Examples (Some example scripts and crontabs for backing up your stuff.) Wiki page edited by grourk - Revision r59 Edited wiki page through web user interface. Revision r59 Edited wiki page through web user interface. Aug 09, 2009 quillen-0.4.zip (v0.4) file uploaded by grourk Aug 09, 2009 r58 (Fix unit tests, prepare for 0.4 release.) committed by grourk - Fix unit tests, prepare for 0.4 release. Fix unit tests, prepare for 0.4 release. Jul 16, 2009 issue 5 (Uppercase bucket names) commented on by jean.jordaan - Just as a data point -- I see this from http://s3tools.org/s3cmd DEBUG: Unicodising "Bucket name 'quillen-1G06DJK4XMS5811GGGR2-chunks' contains disallowed character 'G'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (-)." using UTF-8 Just as a data point -- I see this from http://s3tools.org/s3cmd DEBUG: Unicodising "Bucket name 'quillen-1G06DJK4XMS5811GGGR2-chunks' contains disallowed character 'G'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (-)." using UTF-8 Jul 12, 2009 r57 (Pair needs to be serializable) committed by grourk - Pair needs to be serializable Pair needs to be serializable Jul 11, 2009 r56 (Query objects need to be serializable) committed by grourk - Query objects need to be serializable Query objects need to be serializable Jul 11, 2009 r55 (Foot in the door! (and testing project/svn setup). Fixing t...) committed by blarko - Foot in the door! (and testing project/svn setup). Fixing typo in README. Foot in the door! (and testing project/svn setup). Fixing typo in README. Jul 11, 2009 r54 (fixing tests. reviewed by grourk) committed by martin.davidsson - fixing tests. reviewed by grourk fixing tests. reviewed by grourk Jul 11, 2009 r53 (Introducing AttributeStorageQuery to represent queries, inst...) committed by grourk - Introducing AttributeStorageQuery to represent queries, instead of passing around query strings, which were SimpleDB specific (leaky abstraction!). Constructing these queries is still pretty awkward, though. Will clean this up. Introducing AttributeStorageQuery to represent queries, instead of passing around query strings, which were SimpleDB specific (leaky abstraction!). Constructing these queries is still pretty awkward, though. Will clean this up. Jun 25, 2009 r52 (Introducing SortedFileList, which is a utility for limiting ...) committed by grourk - Introducing SortedFileList, which is a utility for limiting the memory usage when building the list of files to backup. Previously this was all pulled into memory and sorted. Now it will dump sorted batches out to disk if the number of files crosses some threshold and the iterator will stream from those batches in a merge-sort fashion. Introducing SortedFileList, which is a utility for limiting the memory usage when building the list of files to backup. Previously this was all pulled into memory and sorted. Now it will dump sorted batches out to disk if the number of files crosses some threshold and the iterator will stream from those batches in a merge-sort fashion. Jun 07, 2009 r51 (Fixing a bug where a file is snapshoted, then copied, then b...) committed by grourk - Fixing a bug where a file is snapshoted, then copied, then both are snapshotted again but happen to straddle shadow file batches. Fixing it by making sure that shadow files don't straddle batches -- by making sure that batches aren't put until all files of a given size have been processed. Fixing a bug where a file is snapshoted, then copied, then both are snapshotted again but happen to straddle shadow file batches. Fixing it by making sure that shadow files don't straddle batches -- by making sure that batches aren't put until all files of a given size have been processed. Jun 03, 2009 r50 (Darn you IntelliJ!) committed by grourk - Darn you IntelliJ! Darn you IntelliJ! Jun 03, 2009 r49 (Using SDB select instead of the old query syntax. Processin...) committed by grourk - Using SDB select instead of the old query syntax. Processing batches of results as they come (by calling a ResultConsumer callback passed by the caller) instead of letting typca save them all up and returning. Couple of other small bugfixes. Using SDB select instead of the old query syntax. Processing batches of results as they come (by calling a ResultConsumer callback passed by the caller) instead of letting typca save them all up and returning. Couple of other small bugfixes. May 31, 2009 r48 (Optimize the case of restoring a zero byte file (just touch ...) committed by grourk - Optimize the case of restoring a zero byte file (just touch the file). Using # instead of - as the snapshot name / date delimiter in the snapshot attribute of shadow files, because # is less likely to appear in the snapshot name. We may consider making the presence of # in a snapshot name illegal just to avoid any possible ambiguity, although it will work fine, just might be less efficient when querying for shadow files by snapshot name. Optimize the case of restoring a zero byte file (just touch the file). Using # instead of - as the snapshot name / date delimiter in the snapshot attribute of shadow files, because # is less likely to appear in the snapshot name. We may consider making the presence of # in a snapshot name illegal just to avoid any possible ambiguity, although it will work fine, just might be less efficient when querying for shadow files by snapshot name. May 31, 2009 r47 (typica upgrade to 1.5.2a) committed by martin.davidsson - typica upgrade to 1.5.2a typica upgrade to 1.5.2a May 26, 2009 r46 (Eliminate some unecessary calls to Attributes.addReplace(), ...) committed by grourk - Eliminate some unecessary calls to Attributes.addReplace(), which, when put to SDB, incur overhead and cost. Also some other bugfixes. Eliminate some unecessary calls to Attributes.addReplace(), which, when put to SDB, incur overhead and cost. Also some other bugfixes. May 25, 2009 r45 (Reduce the number of workers for batch-putting shadow files,...) committed by grourk - Reduce the number of workers for batch-putting shadow files, since SDB barfs when there's too much concurrency. Also a small bugfix in ChunkServiceImpl. Reduce the number of workers for batch-putting shadow files, since SDB barfs when there's too much concurrency. Also a small bugfix in ChunkServiceImpl. May 25, 2009 r44 (adding target to svn:ignore properties) committed by martin.davidsson - adding target to svn:ignore properties adding target to svn:ignore properties May 21, 2009 r43 (Refactors, cleanups, comments, and a lot of bugfixes.) committed by grourk - Refactors, cleanups, comments, and a lot of bugfixes. Refactors, cleanups, comments, and a lot of bugfixes. May 19, 2009 issue 5 (Uppercase bucket names) commented on by grourk - That is a good idea. I will update the front page with such a warning. I will also throw up a roadmap at some point. Although I've been working on this as a side project when time permits. That is a good idea. I will update the front page with such a warning. I will also throw up a roadmap at some point. Although I've been working on this as a side project when time permits. May 19, 2009 issue 5 (Uppercase bucket names) commented on by jean.jordaan - Ouch. You know, if there are other people using this, it might be worthwhile to check how much this would cost them in transfer and storage costs. Perhaps it's enough to pay you to provide migration instead of paying Amazon and ISPs. Just a thought. Also, since starting over does cost money, it might be worthwhile to put on the frontpage for potential users to consider before they push up 100s of GB. Do you have a roadmap for when the store will stop being throwaway? Ouch. You know, if there are other people using this, it might be worthwhile to check how much this would cost them in transfer and storage costs. Perhaps it's enough to pay you to provide migration instead of paying Amazon and ISPs. Just a thought. Also, since starting over does cost money, it might be worthwhile to put on the frontpage for potential users to consider before they push up 100s of GB. Do you have a roadmap for when the store will stop being throwaway? May 19, 2009 issue 5 (Uppercase bucket names) commented on by grourk - Unfortunately previous versions will need to be uninstalled (all buckets and domains deleted) and re-installed (new buckets and domains created). I try to avoid these kinds of changes, but while the project is this young (still version 0.XX) it is inevitable. Unfortunately previous versions will need to be uninstalled (all buckets and domains deleted) and re-installed (new buckets and domains created). I try to avoid these kinds of changes, but while the project is this young (still version 0.XX) it is inevitable. May 19, 2009 issue 5 (Uppercase bucket names) commented on by jean.jordaan - I really hope it won't be necessary to trash existing backups .. I really hope it won't be necessary to trash existing backups .. May 19, 2009 issue 5 (Uppercase bucket names) Status changed by grourk - Interesting. I will lowercase the bucket names in the next version (which will have a lot of other "breaking changes" that will be incompatible with previous versions. Status: Started Interesting. I will lowercase the bucket names in the next version (which will have a lot of other "breaking changes" that will be incompatible with previous versions. Status: Started May 13, 2009 issue 5 (Uppercase bucket names) reported by jean.jordaan - I tried to access the quillen store using boto http://code.google.com/p/boto/ It failed with: "<class 'boto.exception.BotoClientError'>: S3Error: Bucket names cannot contain upper-case characters when using either the sub-domain or virtual hosting calling format." From looking at the access key, it seems like it can be lowercased without losing information. This would allow more ways of accessing the data. Amazon's docs also indicate that buckets stored in the EU may not contain uppercase characters: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?VirtualHosting.html I tried to access the quillen store using boto http://code.google.com/p/boto/ It failed with: "<class 'boto.exception.BotoClientError'>: S3Error: Bucket names cannot contain upper-case characters when using either the sub-domain or virtual hosting calling format." From looking at the access key, it seems like it can be lowercased without losing information. This would allow more ways of accessing the data. Amazon's docs also indicate that buckets stored in the EU may not contain uppercase characters: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?VirtualHosting.html May 13, 2009 issue 4 (Moving store to a different S3 account) commented on by jean.jordaan - > Just curious: why have the AWS key in the bucket name? Answering myself: because all buckets in S3 share one namespace, across accounts. > Just curious: why have the AWS key in the bucket name? Answering myself: because all buckets in S3 share one namespace, across accounts. May 13, 2009 issue 4 (Moving store to a different S3 account) commented on by jean.jordaan - I followed the instructions here: http://blog.cloudberrylab.com/2009/02/how-to-copy-files-between-amazon-s3.html to copy files to the second S3 account. I'm running CloudBerry S3 Explorer from XP running as a virtual machine. It has some downsides, though. I'm struggling to figure out whether everything was copied OK, and I haven't copied the SDB domain yet. At this point I think I'll start from scratch on the second account. Just curious: why have the AWS key in the bucket name? I followed the instructions here: http://blog.cloudberrylab.com/2009/02/how-to-copy-files-between-amazon-s3.html to copy files to the second S3 account. I'm running CloudBerry S3 Explorer from XP running as a virtual machine. It has some downsides, though. I'm struggling to figure out whether everything was copied OK, and I haven't copied the SDB domain yet. At this point I think I'll start from scratch on the second account. Just curious: why have the AWS key in the bucket name? Apr 12, 2009 r42 (Lots of changes and optimizations: batching up shadow files ...) committed by grourk - Lots of changes and optimizations: batching up shadow files during backup, only getting necessary attributes when possible, moving away from the auto-batching in SDBAttributeStorageImpl and putting the burden onto the client code (ShadowFileServiceImpl, ChunkServiceImpl), using a work queue during install and uninstall, etc. Still a lot to do. Also the unit tests are compiling but probably not passing, and certainly not covering a lot of the new code (yet). Lots of changes and optimizations: batching up shadow files during backup, only getting necessary attributes when possible, moving away from the auto-batching in SDBAttributeStorageImpl and putting the burden onto the client code (ShadowFileServiceImpl, ChunkServiceImpl), using a work queue during install and uninstall, etc. Still a lot to do. Also the unit tests are compiling but probably not passing, and certainly not covering a lot of the new code (yet). Apr 09, 2009 issue 4 (Moving store to a different S3 account) Status changed by grourk - Unfortunately, Quillen does not have support for this right now. You may need to uninstall Quillen on the current account (which will remove its buckets and SDB domains), along with all your data... And then install and re-backup onto your new account. There might be other tools for copying S3 buckets and SimpleDB domains. One that could run on EC2 would probably be ideal. If you use one of them, Quillen's S3 buckets contain the AWS Access Key ID in their names, so you'll have to name the new buckets accordingly. Once you have a copy, you can change the AWS credentials in quillen.properties and Quillen should be none the wiser. Then you can delete the buckets and domains on your old account. Status: Done Unfortunately, Quillen does not have support for this right now. You may need to uninstall Quillen on the current account (which will remove its buckets and SDB domains), along with all your data... And then install and re-backup onto your new account. There might be other tools for copying S3 buckets and SimpleDB domains. One that could run on EC2 would probably be ideal. If you use one of them, Quillen's S3 buckets contain the AWS Access Key ID in their names, so you'll have to name the new buckets accordingly. Once you have a copy, you can change the AWS credentials in quillen.properties and Quillen should be none the wiser. Then you can delete the buckets and domains on your old account. Status: Done Apr 09, 2009 issue 4 (Moving store to a different S3 account) reported by jean.jordaan - I've backed up a lot of data to one AWS account. Now I want to move it to a different account, without transferring all the data again. So I want to move the S3 and SimpleDB stores. Is this practical? I've backed up a lot of data to one AWS account. Now I want to move it to a different account, without transferring all the data again. So I want to move the S3 and SimpleDB stores. Is this practical? Apr 08, 2009 issue 3 (Unexpected response code 500, expected 200) commented on by jean.jordaan - Cool, it looks like I'm set :-) zope@server:/home/zope/local/quillen$ /usr/lib/jvm/jre1.6.0_13/bin/java -jar quillen-0.3.jar -command list [20090408070406] 0 0 Wed Apr 08 07:04:09 UTC 2009 [20090408070530] 0 0 Wed Apr 08 07:05:32 UTC 2009 [20090408114719] 52 72615783519 Wed Apr 08 14:21:09 UTC 2009 Cool, it looks like I'm set :-) zope@server:/home/zope/local/quillen$ /usr/lib/jvm/jre1.6.0_13/bin/java -jar quillen-0.3.jar -command list [20090408070406] 0 0 Wed Apr 08 07:04:09 UTC 2009 [20090408070530] 0 0 Wed Apr 08 07:05:32 UTC 2009 [20090408114719] 52 72615783519 Wed Apr 08 14:21:09 UTC 2009 Apr 08, 2009 issue 3 (Unexpected response code 500, expected 200) Status changed by grourk - Yes. Those messages are warnings from jets3t (the S3 library I use) that it got a server error response from Amazon or encountered some other network hiccup. It retries the request, and after a few tries if it still is unsuccessful then it will error out. Those are "normal" in the sense that they're not unusual and usually it recovers just fine. When the backup is complete, you can double check that it was successful by running the list command and ensuring that the snapshot is listed along with the file count and byte count. If it shows a zero file count and zero byte count, then the snapshot did not finish, in which case you can re-run the command and it should pick up from where it left off. Hope this helps! Status: Done Yes. Those messages are warnings from jets3t (the S3 library I use) that it got a server error response from Amazon or encountered some other network hiccup. It retries the request, and after a few tries if it still is unsuccessful then it will error out. Those are "normal" in the sense that they're not unusual and usually it recovers just fine. When the backup is complete, you can double check that it was successful by running the list command and ensuring that the snapshot is listed along with the file count and byte count. If it shows a zero file count and zero byte count, then the snapshot did not finish, in which case you can re-run the command and it should pick up from where it left off. Hope this helps! Status: Done Apr 07, 2009 issue 3 (Unexpected response code 500, expected 200) commented on by jean.jordaan - (Sorry .. meant this to be "Type-Support", and would have sent to a mailing list if I could find one.) (Sorry .. meant this to be "Type-Support", and would have sent to a mailing list if I could find one.) Apr 07, 2009 issue 3 (Unexpected response code 500, expected 200) reported by jean.jordaan - I'm seeing many stanzas like this repeated during my initial backup -- what does it mean? Is backup progressing? Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RestS3Service performRequest WARNING: Response '/quillen-1G06DJK4XMS5811GGGR2-chunks/8c465a2d39eb2def5aa3e43c59ecb19c65a8c55f-136918' - Unexpected response code 500, expected 200 Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RestS3Service performRequest WARNING: Response '/quillen-1G06DJK4XMS5811GGGR2-chunks/8c465a2d39eb2def5aa3e43c59ecb19c65a8c55f-136918' - Received error response with XML message Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 136981 bytes I'm seeing many stanzas like this repeated during my initial backup -- what does it mean? Is backup progressing? Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RestS3Service performRequest WARNING: Response '/quillen-1G06DJK4XMS5811GGGR2-chunks/8c465a2d39eb2def5aa3e43c59ecb19c65a8c55f-136918' - Unexpected response code 500, expected 200 Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RestS3Service performRequest WARNING: Response '/quillen-1G06DJK4XMS5811GGGR2-chunks/8c465a2d39eb2def5aa3e43c59ecb19c65a8c55f-136918' - Received error response with XML message Apr 8, 2009 8:02:45 AM org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 136981 bytes Apr 01, 2009 r41 (If there's an IOException while copying to a temp file, just...) committed by grourk - If there's an IOException while copying to a temp file, just press on with the original file. If there's an IOException while copying to a temp file, just press on with the original file. Mar 31, 2009 r40 (Create a temporary copy of the file to work with, in case th...) committed by grourk - Create a temporary copy of the file to work with, in case the file changes underneath us. Create a temporary copy of the file to work with, in case the file changes underneath us. Mar 30, 2009 2 issues changed by grourk - Issues 1, 2 Issues 1, 2 Mar 26, 2009 Examples (Some example scripts and crontabs for backing up your stuff.) Wiki page edited by grourk Mar 24, 2009 r38 (0.3-SNAPSHOT -> 0.3) committed by grourk - 0.3-SNAPSHOT -> 0.3 0.3-SNAPSHOT -> 0.3 Mar 24, 2009 quillen-0.3.zip (v0.3) file uploaded by grourk Mar 23, 2009 Examples (Some example scripts and crontabs for backing up your stuff.) Wiki page edited by grourk

issue 8 (org.apache.commons.httpclient.HttpMethodDirector executeWith...) reported by xpd259 - What steps will reproduce the problem? 1. java -jar quillen-0.4.jar -command backup -base ~/Pictures -snapshot pics-`date +%Y` 2> quillen.log 2. files upload 3. What is the expected output? What do you see instead? just a list of files uploaded INFO: backed up file Photos/2009/09/20/dscf2254.jpg INFO: backed up file Photos/2009/10/22/DSCF2420.JPG INFO: backed up file 2009/04/20/DSCF1142.JPG INFO: backed up file Photos/2009/09/20/dscf2236.jpg INFO: backed up file 2008/11/08/dscf0978.jpg 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server sdb.amazonaws.com failed to respond 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request 30-Nov-2009 21:20:06 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:20:07 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 204519 bytes 30-Nov-2009 21:21:05 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:21:06 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 49146 bytes What version of the product are you using? On what operating system? 0.4 ubuntu linux 9.10 Please provide any additional information below. I am trying to upload all my ~/Pictures and it is =>16GB it's hard to tell with other applications using the net but my upload data seems to carry on at 181kb/s even after this error but no matter how many hours I leave it I never get more listings of files been uploaded

What steps will reproduce the problem? 1. java -jar quillen-0.4.jar -command backup -base ~/Pictures -snapshot pics-`date +%Y` 2> quillen.log 2. files upload 3. What is the expected output? What do you see instead? just a list of files uploaded INFO: backed up file Photos/2009/09/20/dscf2254.jpg INFO: backed up file Photos/2009/10/22/DSCF2420.JPG INFO: backed up file 2009/04/20/DSCF1142.JPG INFO: backed up file Photos/2009/09/20/dscf2236.jpg INFO: backed up file 2008/11/08/dscf0978.jpg 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server sdb.amazonaws.com failed to respond 30-Nov-2009 21:09:56 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request 30-Nov-2009 21:20:06 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:20:07 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 204519 bytes 30-Nov-2009 21:21:05 org.jets3t.service.S3Service sleepOnInternalError WARNING: Encountered 1 S3 Internal Server error(s), will retry in 50ms 30-Nov-2009 21:21:06 org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity writeRequest WARNING: Repeating transmission of 49146 bytes What version of the product are you using? On what operating system? 0.4 ubuntu linux 9.10 Please provide any additional information below. I am trying to upload all my ~/Pictures and it is =>16GB it's hard to tell with other applications using the net but my upload data seems to carry on at 181kb/s even after this error but no matter how many hours I leave it I never get more listings of files been uploaded

Last 30 days

Earlier this year