Multipart Upload to Amazon S3 in three steps.

1. Initiate multipart Upload

Post request headers:

POST /ObjectName?uploads HTTP/1.1

Host: BucketName.s3.amazonaws.com

Date: date

Authorization: signatureValue

Data that to be encoded with secret key

final StringBuffer buf = new StringBuffer();

buf.append("POST").append("\n");

buf.append("\n");

buf.append("\n");

buf.append(date).append("\n");

buf.append("/" + bucket + key + "?uploads");

return buf.toString();

Java code snippet to set header:

final PostMethod filePost1 = new PostMethod("http://" + bucket + "." + "s3.amazonaws.com");

filePost1.setPath(key + "?uploads");

filePost1.addRequestHeader("Authorization", "AWS " + awsAccessKeyId + ":" + signature);

filePost1.addRequestHeader("Date", date);

Note: No data to post.

Response

<?xml version="1.0" encoding="UTF-8"?> 

<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">   

      <Bucket>example-bucket</Bucket>

      <Key>example-object</Key>

   <UploadId>

 VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA

      </UploadId> 

</InitiateMultipartUploadResult>

Extract UploadId from response xml, which is needed while uploading parts

2. Upload parts

Post request headers:

PUT /ObjectName?partNumber=PartNumber&uploadId=UploadId HTTP/1.1

Host: BucketName.s3.amazonaws.com 

Date: date 

Content-Length: Size 

Authorization: Signature

Data that to be encoded with secret key

final StringBuffer buf = new StringBuffer();

buf.append("PUT").append("\n");

buf.append("\n");

buf.append("\n");

buf.append(date).append("\n");

buf.append("/" + bucket + key + "?partNumber=" + String.valueOf(partnumber) + "&uploadId=" + uploadID);

return buf.toString();

Java code snippet to set header:

final PutMethod filePut = new PutMethod("http://" + bucket + "." + "s3.amazonaws.com");

filePut.setPath(key + "?partNumber=" + String.valueOf(partNumber) + "&uploadId=" + uploadId);

filePut.addRequestHeader("Authorization", "AWS " + awsAccessKeyId + ":" + signature);

filePut.addRequestHeader("Date", date);

...

final int dateRead = randomAccess.read(buffer, 0, upLoadlimit);

filePut.addRequestHeader("Content-Length", String.valueOf(dateRead));

Note: post part of file.

Response

Extract ETag header form response header

HTTP/1.1 200 OK 

x-amz-id-2: Vvag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg== 

x-amz-request-id: 656c76696e6727732072657175657374 

Date:  Mon, 1 Nov 2010 20:34:56 GMT 

ETag: "b54357faf0632cce46e942fa68356b38" 

Content-Length: 0 

Connection: keep-alive 

Server: AmazonS3

We need map of partid and ETag of all parts to Complete multipart upload

Java code snippet:

final Header etag = filePut.getResponseHeader("ETag");

if ( null != etag )

{

     upTag = etag.getValue();

}

3. Complete Multipart Upload

Post request headers:

POST /ObjectName?uploadId=UploadId HTTP/1.1 

Host: BucketName.s3.amazonaws.com 

Date: Date 

Content-Length: Size 

Authorization: Signature

Data that to be encoded with secret key

final StringBuffer buf = new StringBuffer();

buf.append("POST").append("\n");

buf.append("\n");

buf.append("\n");

buf.append(date).append("\n");

buf.append("/" + bucket + key + "?uploadId=" + uploadID);

return buf.toString();

Java code snippet to set header:

final PostMethod filePost1 = new PostMethod("http://" + bucket + "." + "s3.amazonaws.com");

filePost1.setPath(key + "?uploadId=" + uploadID);

filePost1.addRequestHeader("Authorization", "AWS " + awsAccessKeyId + ":" + signature);

filePost1.addRequestHeader("Date", date);

filePost1.addRequestHeader("Content-Length", String.valueOf(postData.length()));

filePost1.setRequestEntity(new ByteArrayRequestEntity(postData.getBytes()));

Note: postData here is XML format as below.

<CompleteMultipartUpload>

   <Part>     

 <PartNumber>1</PartNumber>      

 <ETag>"a54357aff0632cce46d942af68356b38"</ETag>   

   </Part>   

   <Part>

 <PartNumber>2</PartNumber>               

 <ETag>"0c78aef83f66abc1fa1e8477f296d394"</ETag>

   </Part>

   <Part>

<PartNumber>3</PartNumber>      

<ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag>

   </Part> 

</CompleteMultipartUpload>

Response

<?xml version="1.0" encoding="UTF-8"?> 

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">   

  <Location>http://Example-Bucket.s3.amazonaws.com/Example-Object</Location>   

  <Bucket>Example-Bucket</Bucket>   

  <Key>Example-Object</Key>   

 <ETag>"3858f62230ac3c915f300c664312c11f-9"</ETag> 

</CompleteMultipartUploadResult>

4. Notes & References

1) Except last part all parts must be greater than 5mb

2) Date format should  "EEE, dd MMM yyyy HH:mm:ss GMT"

E.g. Sun, 05 Aug 2007 15:57:11 GMT

References

1) http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadInitiate.html

2) http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

3) http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html

Function used to Encode data with Secret key.

public static String sign(final String data, final String key) throws SignatureException

{

     String result;

     try

     {

         final String HMAC_SHA1_ALGORITHM = "HmacSHA1";

         // get an hmac_sha1 key from the raw key bytes

         final SecretKeySpec signingKey1 = new SecretKeySpec(key.getBytes(), HMAC_SHA1_ALGORITHM);

              // get an hmac_sha1 Mac instance and initialize with the signing key

         final Mac mac1 = Mac.getInstance(HMAC_SHA1_ALGORITHM);

         mac1.init(signingKey1);

              // compute the hmac on input data bytes                         final byte[] rawHmac = mac1.doFinal(data.getBytes());

         // base64-encode the hmac

         result = Base64.encodeBase64String(rawHmac);

         result = result.trim();

     }

     catch ( final Exception e )

     {

         throw new SignatureException("Failed to generate HMAC : " + e.getMessage());

     }

return result;

}

Get src.