Description

baltic-avenue is an open source clone of the Amazon S3 REST API that runs on Google App Engine

Purpose

This project allows you to host a "private instance of S3" on top of Google's infrastructure (big table, etc), leveraging existing client S3 libraries and applications - no need to reinvent the wheel.

Getting Started

Download the project (or get the very latest from source) and run your own instance via the SDK dev server or on production GAE (see FrequentlyAskedQuestions: How do I host my own instance?)

or

Try out the public sandbox instance (see ClientSpecificConfiguration)

• host: http://baltic-avenue.appspot.com
• access key id: test1
• secret key: password1

Status

Although the core S3 operations are implemented, this project is still very much in the proof-of-concept phase. i.e. ...

• Make It Work ← still here
• Make It Right
• Make It Fast

Currently Implemented

• GET service (list buckets)

• PUT/DELETE/HEAD bucket (create/delete bucket)
• GET/PUT bucket acl
• GET bucket (list bucket contents, including delimiter requests where delimiter = '/')

• PUT/DELETE object (create/delete objects)
• GET/HEAD object (get object contents and metadata)

• Multiple user accounts per instance
• User-based ACL operations (and "All Users" group to support public reads)
• Log record info captured and saved to datastore
• Realistic-looking error responses

Limitations

• No ssl/https (app engine issue)
• Objects limited to 1MB (app engine issue)
• No bittorrent
• No vhost-style requests (app engine issue)
• Content-Encoding (and Content-Length) headers can be saved, but not returned (app engine issue)

Still Pending

• Support authenticated-read canned-access policy and authenticated-users group
• Query string authentication
• Request time too skewed
• COPY object
• Testing on non-ascii (and difficult chars) in bucket/keys
• Conditional gets/range queries
• POST object
• Testing on concurrent access/transactions+entity groups
• Performance tweaks - delayed imports and memcache
• Location constraints?
• Bucket access logging? (once/if GAE supports async processing)
• Include object-size in log-record

Client Support

The following S3 client libraries/apps can be configured to use a baltic-avenue instance (see ClientSpecificConfiguration)

• CodePlex.Resourceful (and CodePlex.SpaceBlock) .net
• jets3t/cockpit java
• boto python
• s3sync.rb ruby

More Info

• There is still quite a bit of development to be done, so any and all contributions/feedback are welcome!
• FrequentlyAskedQuestions
• TechnicalDocumentation