What is GoogleFileService?

GoogleFileService is based on Google App Engine for Java, and its aim is to provide data models and APIs for uploading and downloading large files of each size is up to 10MB [1] to and from Google Datastore via HTTPS securely.

Please visit the latest demo on Google App Engine or download the Zip files of source codes to develop or run it yourself.

Key Features (What is New?)

Jun 05, 2009. GoogleFileService v0.1

• To break down barriers of the 1MB size limitation of each Google datastore entity, and extend singal file size up to 10MB.
• To send correct UTF-8 downloaded file name for different browsers such as IE, Firefox, and Google Chrome.
• To upload and download files via HTTPS securely.

Jun 07, 2009. GoogleFileService v0.2

• Add web management UI (Two roles as administrator or regular user. Only administrators can upload files.)
• Add uploading progress bar. [2]

Nov 05, 2009. GoogleFileService v0.3

• Add the setting in 'appengine-web.xml' to determine if only administrators can upload files. If not, then logged in users can just upload and delete their own files, while administrators can upload and delete files belong to any users.
• Fix the bug of getGoogleFileById() to be compatible with the latest SDK appengine-java-sdk-1.2.6. (Thanks for Zhang Yu's help)

Sep 19, 2010. GoogleFileService v0.4

• Upgrade to Apache-HttpClient 4.0.2 and the latest SDK appengine-java-sdk-1.3.7.
• Fix the bug of the escaped chars

Installation

1. Download the latest GoogleFileService Zip files of source codes.
2. Locate GoogleFileService\war\WEB-INF\appengine-web.xml and modify the property 'upload.allowed-client-ip' value to be a list of IPs that can upload files via Apache HttpClient API (if required).
3. Apply your account on Google App Engine for Java.
4. Upload GoogleFileService source codes to Google App Engine for Java. (How to upload)
5. Finally, you can try it on https://[your_app_id].appspot.com/

Getting Started

You can upload your file via the webpage's form or via Apache HttpClient API.

We handle uploaded file by using FileUploadServlet.java defined in web.xml as follows:

<servlet>
    <servlet-name>upload</servlet-name>
    <servlet-class>sinica.googlefileservice.server.servlet.FileUploadServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>upload</servlet-name>
    <url-pattern>/upload</url-pattern>
</servlet-mapping>

1. Webpage's form

If you want to upload files via the webpage, please read as follows:

• Ensure the field names 'fileOwner', 'fileId', 'upfile' all exist, and the 'upfile' is the last one of these three fields to work correctly with Apache Commons FileUpload Streaming API.
• [Notice] Only logged in users can upload files.

<form action="/upload" method="post" enctype="multipart/form-data">
  <input type="hidden" name="fileOwner" />
  <div>FileId:　　<input type="text" name="fileId" size="50" /></div>
  <div>FileObject: <input type="file" name="upfile" size="50" /></div>
  <div><input type="submit" name="submit" value="Upload" /></div>
</form>

2. Apache HttpClient

If you want to upload files via other java application, please read as follows:

• You can use Apache 'httpclient-4.0.2.jar' to send HTTP POST data to Servlet as we do with the webpage's form.
• We write a convenient FileServiceClient.java to handle all the things for sending HTTP POST data, so you just need to create an instance of FileServiceClient and invoke submit() method.
• [Notice] Only allowed client IPs can upload files via Apache HttpClient API. Change WEB-INF/appengine-web.xml if required.

// Google App Engine URL
String url = "https://[your_app_id].appspot.com/upload";		
// Create an instance of FileServiceClient for Google App Engine
FileServiceClient client = FileServiceClientFactory.getFileServiceClient(url);
// submit(String fileId, String fileOwner, String fileName, byte[] fileData, String contentType)
int fileSize = client.submit(fileId, fileOwner, fileName, fileData, contentType);

How It Works

With GoogleFile and GoogleUnit data models (datastore), you can upload a large file of size up to 10 MB very easily via our static method DatastoreUtils.insertGoogleFile().

Some code of FileUploadServlet.java is as follows:

String fileId = "";
String fileOwner = "";
String fileName = "";
int fileSize = -1;
String contentType = "";
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload();
// Set overall request size constraint: the default value of -1 indicates that there is no limit.
upload.setSizeMax(10240000); //10MB, the maximum allowed size, in bytes.
// Set the UTF-8 encoding to grab the correct uploaded filename, especially for Chinese
upload.setHeaderEncoding("UTF-8");
// Parse the request
FileItemIterator iter = upload.getItemIterator(req);
while (iter.hasNext()) {
    FileItemStream item = iter.next();
    InputStream stream = item.openStream();
    String fieldName = item.getFieldName();
    if (item.isFormField()) {
    	//process a regular form field
        if (fieldName.equals("fileId"))
            //set the UTF-8 encoding to grab the correct string
            fileId = Streams.asString(stream, "UTF-8");
        if (fieldName.equals("fileOwner"))
            //set the UTF-8 encoding to grab the correct string
            fileOwner = Streams.asString(stream, "UTF-8");
    } else {
    	//process a file upload
        fileName = item.getName();
        if (fileName != null)
    	    fileName= FilenameUtils.getName(fileName);
        contentType = item.getContentType();
        if (fieldName.equals("upfile")) {
            // Check if the fileId conforms to the Key format of the Google datastore 
            // and all other uploaded fields are not empty.
            if (DatastoreUtils.isKey(fileId) && fileOwner.length() > 0 && fileName.length() > 0) {
                // Save into Google datastore
        	fileSize = DatastoreUtils.insertGoogleFile(fileId, fileOwner, fileName, contentType, stream);
            }
        }
    }
}

Known Issues

1. If we submit a large file of size over 10 MB, then we'll receive a HTTP Error 413 Request Entity Too Large (Your client issued a request that was too large) from Google App Engine.
2. The uploading progress bar works on local development server, but not works on Google App Engine.

Questions or Suggestions?

=> GoogleFileService