From kenneth....@gmail.com on October 22, 2009 04:51:18

Just back from vacation, but I did have a look at the SkyDrive.
Unfortunately there is no official API for SkyDrive, and the "Terms of service"
states that you may only use official tools to access the SkyDrive.

There exists a .Net library here: http://skydriveapiclient.codeplex.com/ The description states:
_At the time of this writing, as far as I know, there is no official API to use for
SkyDrive. SkyDrive Client uses the same communication protocols as a web browser:
it's requesting HTML pages, parsing them (thanks Html Agility Pack project) and
posting back HTML forms filled appropriately. Consequently, SkyDrive WebClient
heavily depends on the design of SkyDrive's HTML pages, even small design or
structural changes could easily break partially or completely this library - please
keep this in mind when you are about to use it in critical/billable projects!. _

So, while it should be easy to develop, it is likely to break and violate the ToS,
which may cause your account to be closed.

Until now, I have specifically avoided developing a GMail backend, because it has the
same drawbacks. If someone else wants to develop and/or support such a backend, feel
free.

If I make this backend, I think it should be an add-on, to be selected during
install, or downloaded seperately, as it may break at any time, should MS decide to
break the API, possibly by accident.

Status: Accepted
Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

From jv13...@gmail.com on May 13, 2011 11:18:52

I might have an acceptable workaround that would require a little bit of manual work once in a while.

The content volumes (I think this is the right word, correct me if I'm wrong) take up the most space. It would make the most sense to upload these to skydrive. What would happen here is, the user would run a backup, then manually upload only the content volumes (preferable with a large volume size to decrease the number of files) to skydrive. The local versions are then deleted. Using the local manifest and signature files, an incremental backup could be done. About once a month or week the user could manualy upload the content volumes to skydrive.

What needs to happen is that duplicati needs to be aware that this is happening. Currently (when using 1.2b) if I make a backup, then delete the content volumes, the next incremental backup will basically be a full backup (it will recreate all the content volumes again). I am pretty sure duplicati also completely deletes the manifest and signature files if there are no corresponding content volumes.

I am not sure if duplicati will be able to detect if a file has changed without the content volumes (just using the manifest and signatures). If the signature or manifest contains a hash for each file, then I think this would be possible.

BTW developers, you guys have done some great work and thank you.

From kenneth@hexad.dk on May 14, 2011 06:37:47

I have seen that someone already maintains an API: http://skydriveapiclient.codeplex.com/ It should be easy to use that as a base for supporting Skydrive as a destination, without requiring that I maintain the html parsing.

Owner: kenneth@hexad.dk
Labels: -Priority-Low Priority-Medium Milestone-Release2.0

From jv13...@gmail.com on May 14, 2011 18:48:53

It would be very neat to use that library, but you don't want a bunch of Microsoft lawyers coming after you later.

From what I described in comment 2, do you think it is possible? I now know that the signature files do contain a hash of each file, so it would be possible to just remove content volumes. Even if it is just a an advanced flag I have to set, I would love to be able to do as described in comment 2.

If this could happen, the content files could be on skydrive, and the signature and manifest could be in dropbox. That way everything will be backed up.

From ernst.m...@gmail.com on May 15, 2011 04:51:49

If You have installed Gladinet Cloud Desktop Starter Edition [1](forever free for Windows XP, Vista, Windows 7, Windows 2003/2008) it is very easy to mount windows Live SkyDrive as a logical local Drive letter, e.g. Drive T.
Si it should be possible to choose a subdirectory of T as a backend for duplicati.
Then Gladinet should sync these backed up files to SkyDrive.

[1] http://www.gladinet.com/c/index.php/gladinet-products-services/

From kenneth@hexad.dk on May 15, 2011 05:11:27

Thanks Ernst, I was about to suggest something like that (I didn't know it was not built in).

@jv13: If I use the library, I am not violating anything (afaik), but the end user would be violating the ToS, which could cause MS to terminate their account.

As for your workaround:
The problem with manual processing is that it can be forgotten. Hence I try to avoid anything that requires repeated human intervention. What would be problematic with having all files on SkyDrive?

The current code tries very hard to detect situations where some files are missing, because it is pretty bad if the backups continue but cannot be restored later. The flag you are asking for would pretty much disable this protection.

From imi...@gmail.com on May 15, 2011 08:54:28

I tried Ernst's suggestion (Gladinet Starter edition).On my system (Win7 - 64 bit) it mounts my SkyDrive storage as Z:\ - a network location. Although this virtual drive location is visible from Windows Explorer, it is not visible from the path browsing function of the duplicati GUI. Is there a way around this?

Also, SkyDrive has 2 policy restrictions that make it an imperfect space for unattended scheduled backup:

(1) You can only upload files, not folders containing files. For those who have data contained in multi-level cascading folders and sub-folders, SkyDrive uploads will fail.

(2) Maximim file size is 50MB. Gladinet has a built-in chunking feature that enables users to break up files into smaller pieces for storage, and so does duplicati. However, until there is a way for duplicati to see the Gladinet network location, this feature is not useful.

Thanks.

From kenneth@hexad.dk on May 15, 2011 09:27:12

You should be able to write the path manually into the dialog. I have tried that with network folders that are not browseable.

From imi...@gmail.com on May 15, 2011 12:45:53

You're 100% correct, Kenneth. At the Gladinet forums I learned that the path to the root of a mounted Gladinet virtual drive is via localhost ... specifically:

\127.0.0.1\resources\

By manually entering the path in this format, duplicati connects to SkyDrive. When time permits, I will test this to see if I can overcome the policy restrictions I noted in Comment #8.

Thanks.

From ernst.m...@gmail.com on May 16, 2011 10:44:55

Gladinet does not sync up the files (zip+manifest) to the Online SkyDrive

I can select all folders of Windows Live SkyDrive on the Gladinet Drive T on my Vista Business 64 bit in normal Windows Explorer as a Duplicati Backend in the Duplicati storage options, see ScreenShot http://is.gd/zzsvaJ .

I have Gladinet 1.3.208.5638 Professional Edition, not Starter Edition, as I thought first.
A sub-sub-folder of BU (=BackUp) should be the Duplicati Backend:
T:\Windows Live SkyDrive\BU\Q6.D.PApps+Sich.Dupl
But Gladinet does not sync up the files (zip+maifest) in Duplicati-Subfolders of T:\Windows Live Skydrive\BU to the Online SkyDrive Storage. All backup-files of Duplicati remain local on my Vista 64bit-PC.

The explanation, that Gladinet does not sync up the Duplicati Files seems to be the information in Comment 7 by imi...@gmail.com:
(1) You can only upload files, not folders containing files. For those who have data contained in multi-level cascading folders and sub-folders, SkyDrive uploads will fail.

But maybe there is a solution to load up backup-files in spite of these policy restrictions.
I tested to create empty folders online on Skydrive and then only to backup files in existing folders:
But unfortunatelly without success: Gladinet did not sync up the local Duplicati zip- and manifest-files.
Best Regards Ernst (Austria)

From kenneth@hexad.dk on May 28, 2011 01:36:41

It sucks a bit to have you Duplicati files in the root of the "T" drive, but Duplicati should be able to handle that.

@Ernst: Does that mean that the files in "2008 München" and the others are not transferred to SkyDrive either?

From imi...@gmail.com on May 28, 2011 14:03:48

I experienced the same problems that Ernst mentioned in post #10. duplicati correctly backs up the files, but Gladinet will not upload them to SkyDrive. They remain cached on my local system. I've tried Gladinet, SDExplorer and had the same poor results.

I conclude that SkyDrive is not a good location for automatic scheduled backup of dynamically changing files. Besides the limitations of 50MB per file, and no folders, I believe that Microsoft has built some other undocumented limitations into SkyDrive that limit its compatibility with conventional backup applications.

However, the available free 25GB is tempting, and I tested using SkyDrive for long-term storage (rather than backup) of static archived files. In this scenario, I have achieved some success, although I have to execute some of the steps manually. Here's how it works:

(1) Using duplicati, I backup the targeted archive folders to a local folder on my system, converting the entire archive into *.zip.aes files, with a maximum volume size of 40MB.

(2) Using SD Explorer (Gladinet and the SkyDrive website will also work), I manually drag and drop all the zip.aes files into a new folder in SkyDrive.

It's slow, it is a little awkward - but it works.

From ernst.m...@gmail.com on May 29, 2011 09:45:08

@kenneth (c11):Does that mean that the files in "2008 München" and the others are not transferred to SkyDrive either?
I tested it and put a sreenshot-png-file with 69 kB in the Folder T:\Windows Live SkyDrive\2008 München. It as the same as with the Duplicati-Files: Gladinet did not upload it/them to SkyDrive. Then I stored the same sreenshot-png-file with 69 kB in T:\Windows Live SkyDrive\BU, but again Gladinet did not upload it to SkyDrive.
I could read a very short error message during the upload attempts:
"Analyze failed upload". Google Search with "Windows Live SkyDrive Gladinet Analyze failed upload" see http://is.gd/TLd3ay . It seems that the upload function of Gladinet to Live SkyDrive generally does not work.

From kenneth....@gmail.com on June 28, 2011 23:30:22

This issue was updated by revision r820 .

There is now a SkyDrive backend.

Status: Fixed

From imi...@gmail.com on June 29, 2011 07:03:28

Error: Invalid URI

I am using r820 (64 bit) and the GUI. duplicati creates a backup folder in my Skydrive account, and "Test Connection" is successful. However, when I complete the Wizard, and run the backup job, it crashes and I receive this error:

Error: Invalid URI: The hostname could not be parsed.

Error: System.UriFormatException: Invalid URI: The hostname could not be parsed.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at Duplicati.Library.Main.Interface.ValidateOptions(CommunicationStatistics stats)
at Duplicati.Library.Main.Interface.SetupCommonOptions(CommunicationStatistics stats)
at Duplicati.Library.Main.Interface.Backup(String[] sources)
at Duplicati.GUI.DuplicatiRunner.ExecuteTask(IDuplicityTask task)
Cleanup output:
Error: System.UriFormatException: Invalid URI: The hostname could not be parsed.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at Duplicati.Library.Main.Interface.ValidateOptions(CommunicationStatistics stats)
at Duplicati.Library.Main.Interface.SetupCommonOptions(CommunicationStatistics stats)
at Duplicati.Library.Main.Interface.DeleteAllButNFull()

at Duplicati.GUI.DuplicatiRunner.ExecuteTask(IDuplicityTask task)

Thanks.

From ernst.m...@gmail.com on June 29, 2011 11:14:37

Unique and very useful: Duplicati can now backup to Windows Live SkyDrive
I use Duplicati 1.2.1 r820 .zip (portable) in German language from https://code.google.com/p/duplicati/downloads/list .
The window (1) "Backup storage options" should be translated to German.
It seems that some strings have been translated already, but are not used in the window in Duplicati 1.2.1 r820 .zip .
Some strings must be added to the translation file report.de-DE (2)
I tested a Duplicati-Backup of a folder with 6.63MB to the in Duplicati created folder Duplicati.BU in my SkyDrive.
To complete the test I restored the files afterwards and tested the integrity. All files are OK. Very perfect!
I know no other backup-app, which is able to do backups to MS Live SkyDrive as Duplicati does.
(1) http://is.gd/9jERu2 (2) http://goo.gl/nX2C9 I posted a copy of this comment on (3) and (4) :
(3) http://alternativeto.net/software/duplicati/comments (4) http://alternativeto.net/software/windows-live-skydrive/comments

From imi...@gmail.com on June 29, 2011 16:02:20

Update to my post # 15:

The problem is intermittent. After I rebooted, I was able to backup to SkyDrive and recover, without error. I don't know which local machine conditions lead to a failed or to a successful backup. However, all fails are "Invalid URI: The hostname could not be parsed."

From imi...@gmail.com on June 30, 2011 07:02:50

I can confirm that r822 has corrected the "Invalid URI" problem with SkyDrive. I tested with several small backup and recovery jobs (<100MB) without errors. Thanks Kenneth!

HOWEVER, not all is perfect with the Duplicati to SkyDrive solution. In one test, I attempted to backup a folder holding a 1GB Truecrypt container - using .zip compression and .aes encryption - all broken into 40MB chunks (volumes). Admittedly, a more extreme test.

This should have resulted in an upload of ~25 volumes. After the 3rd volume, I received the following error:

"The remote server returned an error: (403) Forbidden."

I am pretty sure this 403 error has nothing to do with Duplicati, and is a result of me violating some unknown SkyDrive API property. It will take trial and error for Duplicati users to discover the limits of backing up to SkyDrive. As long as we stay inside those limits, we'll probably be OK (assuming SkyDrive doesn't modify the API again!).

Thank you.

From kenneth@hexad.dk on July 01, 2011 02:55:21

The comment on SkyDrive that "you may violate the ToS" is simply a reference to the ToS for SkyDrive that states that "you may only use official API to access ...", and since there is no official (public) API, you are probably violating that rule by using Duplicati to access SkyDrive.

The 403 may be because it authenticates too often. It will re-authenticate for each request, which may cause the SkyDrive server to think it is under attack or similar, although I have not seen it in my tests. This will be fixed in the next final release.

Is the problem repeatable? If you can get it to give the 403 by trying the same load again, I can try to replicate and see if I can fix it.

From imi...@gmail.com on July 02, 2011 03:59:30

RE: 403 Error:

I am traveling, and away from my test machine. I will re-try the upload in 3 or 4 days to determine if the problem is repeatable - and let you know.
Thx.

From imi...@gmail.com on July 05, 2011 05:49:43

The problem is repeatable. I re-tested several times, and I receive the 403 error 100% of the time.

If you would like to replicate, here are the parameters:
(1) On local drive, create a folder containing a 1GB Truecrypt container
(2) Using duplicati UI, create new Backup job from local source to a newly created folder in SkyDrive
(3) .zip + .aes
(4) Volume size = 40MB

Thanks.

From bobbalz...@gmail.com on October 03, 2011 19:51:21

The 403 (forbidden) error is caused by an attempt to update an existing file (i.e. upload a new version when a file of that name exists in the upload folder). This happens with Duplicati when uploading the manifest of the third volume (because
Duplicati switches back and forth between the manifestA and manifestB names so
the third volume causes the first attempt to update an existing file).

The problem is readily solved by deleting the existing file before uploading the new version.

This can be accomplished with the following changes to SkyDrive.CS:

1. Add to SkyDrive class:
   private List m_uploadedFileNames = new List();

2. Replace Put method in SkyDrive class by:

   public void Put(string remotename, System.IO.Stream stream)
   {
   using (SkyDriveSession session = CreateSession(false)){
   bool newFileBeingUploaded;
   if (m_uploadedFileNames.Contains(remotename)){ // file already uploaded. Delete it before uploading replacement
   try {
   HttpWebResponse deleteFileResponse = session.DeleteFile(remotename); // delete already existing file
   } catch (Exception ex){ // ignore any exceptions arising from DeleteFile
   System.Diagnostics.Debug.WriteLine("Ignoring DeleteFile error " + ex.Message + " for file " + remotename);
   }
   newFileBeingUploaded = false;
   } else {
   newFileBeingUploaded = true;
   }
   using (System.Net.HttpWebResponse resp = session.UploadFile(remotename, stream)){
   int code = (int)resp.StatusCode;
   if (code < 200 || code >= 300) //For some reason Mono does not throw this automatically
   throw new System.Net.WebException(resp.StatusDescription, null, System.Net.WebExceptionStatus.ProtocolError, resp);
   }
   if (newFileBeingUploaded){
   m_uploadedFileNames.Add(remotename); // record newly uploaded file
   }
   }
   }

From kenneth@hexad.dk on October 04, 2011 01:12:24

Arh, of course.

The delete/upload is two steps, and something could go wrong after the delete.
I will try to see if I can fix it so it overwrites the file instead.

Labels: -Milestone-Release2.0 Milestone-Release1.3

From kenneth@hexad.dk on October 16, 2011 09:32:26

Status: Started

From kenneth@hexad.dk on October 16, 2011 09:33:24

Issue 459 has been merged into this issue.

From kenneth....@gmail.com on October 25, 2011 12:08:07

This issue was updated by revision r945 .

I managed to figure out how to overwrite existing files, instead of the delete-then-upload fix.
I also fixed some other minor issues with SkyDrive.

A new preview release will be up asap.

Status: Fixed

From imi...@gmail.com on October 25, 2011 15:16:43

Confirmed as fixed! Using 1.2.3 r946 (64 bit) I was able to backup a 7 volume job to SkyDrive without any errors, and it restored back to my system accurately.

Congratulations - I believe that duplicati is the only backup software ANYWHERE with this capability, and thank you!

From kenneth@hexad.dk on June 04, 2012 12:30:28

Issue 639 has been merged into this issue.