timedog - Project Hosting on Google Code

Older Nov 29, 2009 issue 7 (Error when executing the script) commented on by jdtsmith - It's possible you have run the command from a directory corresponding to a volume which has been unmounted (e.g. the TimeMachine volume). It's possible you have run the command from a directory corresponding to a volume which has been unmounted (e.g. the TimeMachine volume). Nov 29, 2009 issue 6 (Run as root) Owner changed by jdtsmith - Owner: jdtsmith Owner: jdtsmith Nov 29, 2009 issue 6 (Run as root) Status changed by jdtsmith - Good point, and worth mentioning. Status: Done Good point, and worth mentioning. Status: Done Nov 12, 2009 issue 7 (Error when executing the script) reported by d...@appcabinet.com - What steps will reproduce the problem? 1. run timedob -d 5 -l What is the expected output? What do you see instead? I am supposed to get some results but all I get is "nvalid top directory at /System/Library/Perl/5.8.8/File/Find.pm line 592." What version of the product are you using? On what operating system? The latest as of now. Just downloaded the file. Please provide any additional information below. OS: X Leopard Backup Device: Time Capsule What steps will reproduce the problem? 1. run timedob -d 5 -l What is the expected output? What do you see instead? I am supposed to get some results but all I get is "nvalid top directory at /System/Library/Perl/5.8.8/File/Find.pm line 592." What version of the product are you using? On what operating system? The latest as of now. Just downloaded the file. Please provide any additional information below. OS: X Leopard Backup Device: Time Capsule Jul 10, 2009 issue 6 (Run as root) reported by avbentem - When not running the current version of timedog as root (so, even when running as an admin user), it silently ignores (or simply doesn't know about) files it cannot access. This also yields a lower total size of changed files. I don't think this is a common problem (though maybe plain users, without admin rights, need sudo all the time), so maybe this could simply be documented. For example, running as an admin user: ~/Downloads/timedog -d 1 ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 35.7MB-> 60.3MB [1416] /Macintosh HD/ ==> Total Backup: 1421 changed files/directories, 60.34MB sudo ~/Downloads/timedog -d 1 Password: ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 14.3GB-> 14.4GB [1423] /Macintosh HD/ ==> Total Backup: 1428 changed files/directories, 14.36GB My backup disk is attached to a remote Mac mini, so it's a sparsebundle. The same issue occurs in TimeTracker. I found it because the logs stated many GBs were used, and disk space did indeed decrease by many GBs. However, timedog and TimeTracker only reported a much lower value. Using "find . -mtime -10" in the latest backup I got a warning: find: ./Macintosh HD/usr/local/mysql-5.0.51a-osx10.5-x86/data: Permission denied And that folder indeed showed a 11 GB log file, and a 3.5 GB database. This folder is owned by user _mysql, group wheel. When not running the current version of timedog as root (so, even when running as an admin user), it silently ignores (or simply doesn't know about) files it cannot access. This also yields a lower total size of changed files. I don't think this is a common problem (though maybe plain users, without admin rights, need sudo all the time), so maybe this could simply be documented. For example, running as an admin user: ~/Downloads/timedog -d 1 ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 35.7MB-> 60.3MB [1416] /Macintosh HD/ ==> Total Backup: 1421 changed files/directories, 60.34MB sudo ~/Downloads/timedog -d 1 Password: ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 14.3GB-> 14.4GB [1423] /Macintosh HD/ ==> Total Backup: 1428 changed files/directories, 14.36GB My backup disk is attached to a remote Mac mini, so it's a sparsebundle. The same issue occurs in TimeTracker. I found it because the logs stated many GBs were used, and disk space did indeed decrease by many GBs. However, timedog and TimeTracker only reported a much lower value. Using "find . -mtime -10" in the latest backup I got a warning: find: ./Macintosh HD/usr/local/mysql-5.0.51a-osx10.5-x86/data: Permission denied And that folder indeed showed a 11 GB log file, and a 3.5 GB database. This folder is owned by user _mysql, group wheel. May 04, 2009 issue 5 (timecopy.py: unexpected files in Backups.backupdb directory) Status changed by nathanfiedler - Fixed in Subversion revision 26. Status: Fixed Fixed in Subversion revision 26. Status: Fixed May 04, 2009 r26 (Issue #5: ignore .DS_Store entries in special directories.) committed by nathanfiedler - Issue #5 : ignore .DS_Store entries in special directories. Issue #5 : ignore .DS_Store entries in special directories. Apr 19, 2009 issue 5 (timecopy.py: unexpected files in Backups.backupdb directory) commented on by nathanfiedler - Should change timecopy.py to be smarter about what to look for in that backupdb directory. Namely, only consider directories and only those whose names do not start with a dot (i.e. ignore hidden files). Should change timecopy.py to be smarter about what to look for in that backupdb directory. Namely, only consider directories and only those whose names do not start with a dot (i.e. ignore hidden files). Apr 19, 2009 issue 5 (timecopy.py: unexpected files in Backups.backupdb directory) reported by nathanfiedler - From michaschnecke: While I am not an expert in Python, I have following directory message, which confuses me: whitebook:Desktop micha$ sudo ./timecopy.py /Volumes/hfs /Volumes/whitebook Traceback (most recent call last): File "./timecopy.py", line 499, in <module> main() File "./timecopy.py", line 493, in main copybackupdb(src, dst, verbose, dryrun, extattr) File "./timecopy.py", line 326, in copybackupdb entries = os.listdir(src) OSError: [Errno 20] Not a directory: '/Volumes/hfs/Backups.backupdb/.com.apple.timemachine.supported' From michaschnecke: While I am not an expert in Python, I have following directory message, which confuses me: whitebook:Desktop micha$ sudo ./timecopy.py /Volumes/hfs /Volumes/whitebook Traceback (most recent call last): File "./timecopy.py", line 499, in <module> main() File "./timecopy.py", line 493, in main copybackupdb(src, dst, verbose, dryrun, extattr) File "./timecopy.py", line 326, in copybackupdb entries = os.listdir(src) OSError: [Errno 20] Not a directory: '/Volumes/hfs/Backups.backupdb/.com.apple.timemachine.supported' Apr 19, 2009 issue 3 (timecopy: script that intelligently copies TM volumes) Status changed by nathanfiedler - This script has been completed for some time now, and has been tested on three different systems. Status: Done This script has been completed for some time now, and has been tested on three different systems. Status: Done Feb 17, 2009 UsingTimecopy (Instructions for using timecopy script.) Wiki page edited by nathanfiedler Feb 16, 2009 UsingTimecopy (Instructions for using timecopy script.) Wiki page edited by nathanfiedler Feb 16, 2009 UsingTimecopy (Instructions for using timecopy script.) Wiki page edited by nathanfiedler Feb 16, 2009 UsingTimecopy (Instructions for using timecopy script.) Wiki page edited by nathanfiedler Feb 16, 2009 UsingTimecopy (Instructions for using timecopy script.) Wiki page added by nathanfiedler Feb 15, 2009 r20 (Use xattr to copy important extended attributes.) committed by nathanfiedler - Use xattr to copy important extended attributes. Use xattr to copy important extended attributes. Feb 15, 2009 r19 (Add missing verbose/dryrun conditionals.) committed by nathanfiedler - Add missing verbose/dryrun conditionals. Add missing verbose/dryrun conditionals. Feb 14, 2009 r18 (A script for comparing two TM volumes.) committed by nathanfiedler - A script for comparing two TM volumes. A script for comparing two TM volumes. Feb 14, 2009 r17 (A completely working TM volume copier.) committed by nathanfiedler - A completely working TM volume copier. A completely working TM volume copier. Feb 10, 2009 r16 (Some fixes in links; skip existing snapshots.) committed by nathanfiedler - Some fixes in links; skip existing snapshots. Some fixes in links; skip existing snapshots. Feb 08, 2009 r15 (Gracefully handle failure to chown links and directories.) committed by nathanfiedler - Gracefully handle failure to chown links and directories. Gracefully handle failure to chown links and directories. Jan 31, 2009 r14 (Another option to not invoke chown; more fixes.) committed by nathanfiedler - Another option to not invoke chown; more fixes. Another option to not invoke chown; more fixes. Jan 27, 2009 r13 (Initial dryrun tests appear to be working.) committed by nathanfiedler - Initial dryrun tests appear to be working. Initial dryrun tests appear to be working. Jan 26, 2009 r12 (Added verbose/dryrun options; more copying logic is in place...) committed by nathanfiedler - Added verbose/dryrun options; more copying logic is in place. Added verbose/dryrun options; more copying logic is in place. Jan 25, 2009 r11 (Partial implementation of timecopy Python script, not yet wo...) committed by nathanfiedler - Partial implementation of timecopy Python script, not yet working. Partial implementation of timecopy Python script, not yet working. Jan 25, 2009 issue 3 (timecopy: script that intelligently copies TM volumes) Status changed by nathanfiedler - An implementation is in progress, written in Python. Status: Started An implementation is in progress, written in Python. Status: Started Jan 23, 2009 issue 4 (Spin new build of latest timedog) Status changed by nathanfiedler - v1.2 released, previous release deprecated. Status: Done v1.2 released, previous release deprecated. Status: Done Jan 23, 2009 timedog-1.2.zip (timedog v1.2) file uploaded by nathanfiedler - Labels: Featured Type-Archive OpSys-OSX Labels: Featured Type-Archive OpSys-OSX Jan 23, 2009 issue 4 (Spin new build of latest timedog) reported by nathanfiedler - Sean has submitted a patch that JD committed in rev 9. Make a new build and put on the downloads page. Sean has submitted a patch that JD committed in rev 9. Make a new build and put on the downloads page. Jan 23, 2009 r10 (Added Id back. ) committed by jdtsmith - Added Id back. Added Id back. Jan 23, 2009 r9 (Added -s for sorting by size, and -m, for ignoring files/dir...) committed by jdtsmith - Added -s for sorting by size, and -m, for ignoring files/directories based on size. --This line, and those below, will be ignored-- M timedog Added -s for sorting by size, and -m, for ignoring files/directories based on size. --This line, and those below, will be ignored-- M timedog Jan 22, 2009 r8 (More user input validation.) committed by nathanfiedler - More user input validation. More user input validation. Jan 22, 2009 r7 (Setting keywords property to Id.) committed by nathanfiedler - Setting keywords property to Id. Setting keywords property to Id. Jan 22, 2009 r6 (Adding Python script that will eventually copy Time Machine ...) committed by nathanfiedler - Adding Python script that will eventually copy Time Machine volumes. Adding Python script that will eventually copy Time Machine volumes. Jan 18, 2009 issue 3 (timecopy: script that intelligently copies TM volumes) reported by nathanfiedler - My TM sparse bundle disk image has an "invalid sibling link" error, and one strategy I'm employing to correct this "unfixable" error is to use the trial version of Super Duper! to copy the TM files over to a new disk image. It seems to be working fine, but it's extremely slow: 100GB in about 36 hours. While Super Duper! is handling the hard links properly, it's spending a lot of time resolving them across the entire file system. It would go a lot faster if it were aware of the TM file structure, as timedog is. In email communication with the SD people, it became clear that they don't see this as a problem. I, however, do. I propose a new script, call it timecopy, that does one thing well: copies a TM volume to another volume. It will traverse the directories in date order and discover the hard links, reproducing the same structure in the new volume, and copying over the files as needed. For bonus points, it should be able to resume from where a previous copy left off. That is, discover that the target volume already has some number of backups and start from where it left off, copying any new backups from the source to the target. My TM sparse bundle disk image has an "invalid sibling link" error, and one strategy I'm employing to correct this "unfixable" error is to use the trial version of Super Duper! to copy the TM files over to a new disk image. It seems to be working fine, but it's extremely slow: 100GB in about 36 hours. While Super Duper! is handling the hard links properly, it's spending a lot of time resolving them across the entire file system. It would go a lot faster if it were aware of the TM file structure, as timedog is. In email communication with the SD people, it became clear that they don't see this as a problem. I, however, do. I propose a new script, call it timecopy, that does one thing well: copies a TM volume to another volume. It will traverse the directories in date order and discover the hard links, reproducing the same structure in the new volume, and copying over the files as needed. For bonus points, it should be able to resume from where a previous copy left off. That is, discover that the target volume already has some number of backups and start from where it left off, copying any new backups from the source to the target. Jan 16, 2009 issue 2 (Auto-discover and mount TM volume) reported by nathanfiedler - JD points out that auto-discovering and mounting the Time Machine volume would be a nice feature. It seems that tms has this, so should be simple enough to see how they do it. JD points out that auto-discovering and mounting the Time Machine volume would be a nice feature. It seems that tms has this, so should be simple enough to see how they do it. Jan 16, 2009 issue 1 (Add instructions for mounting a network backup) Status changed by nathanfiedler - Added some initial documentation. Status: Fixed Added some initial documentation. Status: Fixed Jan 15, 2009 r5 (Set EOL style to native, just for completeness.) committed by nathanfiedler - Set EOL style to native, just for completeness. Set EOL style to native, just for completeness. Jan 15, 2009 issue 1 (Add instructions for mounting a network backup) reported by nathanfiedler - For people that use Time Capsule, they may be unaware of how to mount their backup image, and until then, timedog won't work for them. Basically it's just a matter of opening Disk Utility and mounting the sparse bundle disk image. For people that use Time Capsule, they may be unaware of how to mount their backup image, and until then, timedog won't work for them. Basically it's just a matter of opening Disk Utility and mounting the sparse bundle disk image. Jan 15, 2009 timedog-1.1.zip (timedog v1.1) file uploaded by nathanfiedler - Labels: Featured Type-Archive OpSys-OSX Labels: Featured Type-Archive OpSys-OSX Jan 15, 2009 r4 (Changes made by JD: adds -n option) committed by nathanfiedler - Changes made by JD: adds -n option Changes made by JD: adds -n option Jan 15, 2009 r3 (Setting keywords property) committed by nathanfiedler - Setting keywords property Setting keywords property Jan 15, 2009 r2 (Initial checkin of original script.) committed by nathanfiedler - Initial checkin of original script. Initial checkin of original script. Jan 15, 2009 Project timedog created by nathanfiedler - View files saved by Time Machine View files saved by Time Machine

issue 6 (Run as root) reported by avbentem - When not running the current version of timedog as root (so, even when running as an admin user), it silently ignores (or simply doesn't know about) files it cannot access. This also yields a lower total size of changed files. I don't think this is a common problem (though maybe plain users, without admin rights, need sudo all the time), so maybe this could simply be documented. For example, running as an admin user: ~/Downloads/timedog -d 1 ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 35.7MB-> 60.3MB [1416] /Macintosh HD/ ==> Total Backup: 1421 changed files/directories, 60.34MB sudo ~/Downloads/timedog -d 1 Password: ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 14.3GB-> 14.4GB [1423] /Macintosh HD/ ==> Total Backup: 1428 changed files/directories, 14.36GB My backup disk is attached to a remote Mac mini, so it's a sparsebundle. The same issue occurs in TimeTracker. I found it because the logs stated many GBs were used, and disk space did indeed decrease by many GBs. However, timedog and TimeTracker only reported a much lower value. Using "find . -mtime -10" in the latest backup I got a warning: find: ./Macintosh HD/usr/local/mysql-5.0.51a-osx10.5-x86/data: Permission denied And that folder indeed showed a 11 GB log file, and a 3.5 GB database. This folder is owned by user _mysql, group wheel.

When not running the current version of timedog as root (so, even when running as an admin user), it silently ignores (or simply doesn't know about) files it cannot access. This also yields a lower total size of changed files. I don't think this is a common problem (though maybe plain users, without admin rights, need sudo all the time), so maybe this could simply be documented. For example, running as an admin user: ~/Downloads/timedog -d 1 ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 35.7MB-> 60.3MB [1416] /Macintosh HD/ ==> Total Backup: 1421 changed files/directories, 60.34MB sudo ~/Downloads/timedog -d 1 Password: ==> Comparing TM backup 2009-07-09-213048 to 2009-07-01-140540 Depth: 1 directories 3.2KB-> 3.2KB /.Backup.log 0B-> 0B /.com.apple.TMCheckpoint 788B-> 788B /.exclusions.plist 14.3GB-> 14.4GB [1423] /Macintosh HD/ ==> Total Backup: 1428 changed files/directories, 14.36GB My backup disk is attached to a remote Mac mini, so it's a sparsebundle. The same issue occurs in TimeTracker. I found it because the logs stated many GBs were used, and disk space did indeed decrease by many GBs. However, timedog and TimeTracker only reported a much lower value. Using "find . -mtime -10" in the latest backup I got a warning: find: ./Macintosh HD/usr/local/mysql-5.0.51a-osx10.5-x86/data: Permission denied And that folder indeed showed a 11 GB log file, and a 3.5 GB database. This folder is owned by user _mysql, group wheel.

Older