Updated Jun 07, 2008 by br...@mcbarron.net
Labels:	DesignDoc

ContentRangePostProposal

A proposal for supporting Content-Range in HTTP POST and PUT requests.

Introduction

The primary primitive necessary to enable large file uploads is the ability to specify byte ranges in POST and PUT requests. Byte ranges are already standardized for GET requests, and there are implementations of byte-range PUT used by WebDAV, but to our knowledge there has been little effort to use them for POST. Byte-range POST/PUT could be used to resume incomplete transfers, or to explicitly break transfers into smaller sized chunks. We propose standardizing byte range POST/PUT in a manner analogous to byte range GET.

The use of Content-Range headers in POST/PUT is allowed by the standard. Section 9.6 explicitly suggests the possibility of using Content-Range headers in PUTs: "The recipient of the entity MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501 (Not Implemented) response in such cases." However, such functionality has not been widely deployed, and therefore there exists no reference implementation or standardized semantics for how it should be used.

Proposal

Content-Range values used for POST/PUT should be used as described by HTTP/1.1 section 14.16 with the following differences:

Clients MUST NOT send a byte-range-resp-spec of "*" when transferring bytes, as the server would have no way to determine the byte range being transferred.
Clients MAY omit last-byte-pos of if the length of the range being transferred is unknown or difficult to determine.
Clients SHOULD specify the instance-length, unless it is unknown or difficult to determine (in which case "*" should be used).

Use Cases

The use of Content-Range POST/PUT could be used to compose a chunked transfer protocol where a large POST/PUT was broken up into many small POST/PUTs. This would be similar to the protocols used by several current resumable uploaders. The basic algorithm for this model is to continue sending each chunk (probably in serial order) until it is successfully acknowledged. Note that if used for POSTs, some server-specific mechanism would be required to uniquely identify the the set of chunks constituting a single logical transfer. If it is necessary to accommodate server-side chunk loss (e.g. due to failures partially masked by replication) some additional custom protocol components would be necessary for the server to indicate to the client which chunks it had successfully received.

For some web applications, it may be desirable to upload a subset of a file to the server to reduce the latency between when a file is selected, and when the user can manipulate it in the application. A concrete example of this might be an image hosting website, which would like to upload the 64KB EXIF data segment from a JPEG file, so that the user can quickly view a thumbnail version of the file in the application while the rest of the image is transferred in the background.

Gears Modifications

It is already feasible to write JavaScript which emits POST or PUT with a Content-Range header, by utilizing the setRequestHeader method of XMLHttpRequest or Gears.HttpRequest. However, those objects currently only support string or XML document payloads, which are not that interesting from our perspective, since they are unlikely to be large enough to warrant segmented upload. There is an existing proposal for supporting upload of files and other Binary Large OBjects via the Gears.Blob API, which is very relevant to this proposal. The Blob API and it's integration with HttpRequest promises a programmatic way to upload large content. To support segmentation, we propose an extension to Gears.Blob which would enable segmentation of an existing Blob. The interface is as follows:

interface Blob {
  ...
  Blob slice(int64 offset);
  Blob slice(int64 offset, int64 length);
  ...
}

The resulting Blob would be equivalent to the target Blob, except that it begin with the data at the byte position offset, and continue for length bytes (or the end of data if not specified).

Utilizing this interface, along with Gears.HttpRequest.send(Blob) and setRequestHeader, one can implement this proposal for the upload of segments of files or other large objects to which the script has access.

Related Work

RFC 2616: HTTP/1.1

http://www.w3.org/Protocols/rfc2616/rfc2616.html

9.6	Method: PUT
14.16	Header Field: Content-Range

Apache mod_dav

http://www.webdav.org/mod_dav/

Apache mod_dav added support for Content-Range PUT's in version 1.0.0-1.3.6, released June 13, 2000.

Comment by electronixtar, Oct 01, 2008

great stuff! I am implementing a multi-threaded uploading app using python+gears

Comment by jlouvel, Oct 20, 2008

We have recently added support for partial and resumable PUT in the Restlet framework (REST framework for Java, http://www.restlet.org).

See design notes: http://wiki.restlet.org/developers/172-restlet/154-restlet.html?branch=docs-1_1&language=en

Best regards, Jerome

Comment by jaakko.manninen, Dec 04, 2008

Fun, thanks! But be careful with caches caching the 'partials' as the 'entire' (or use no-cache headers), as this is non-standard for PUT and can be dangerous, see: http://markmail.org/message/6jyvhg4swcusyvn5 - use PATCH if you can instead

Comment by todd.fisher, Feb 17, 2009

How is this related to or different from the proposal here: http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal ?