Step-by-Step Guide to Uploading and Downloading Big Files to S3

Here, I am going to talk about the process to upload large objects to Amazon S3 using the multipart approach.

The multipart approach consists of 3 steps:

1. Initiate Multipart Upload

2. Parts Upload

3. Complete Upload

Let's discuss these steps in detail.

1. **Initiate Multipart Upload**:

To initiate a large object upload, the client sends a request to the S3 server. The server responds with an upload ID, which is unique to each upload.

2. **Parts Upload**:

To start uploading the object in parts, you must have the upload ID from the previous step and a part number in incremental or sequential order. The part number identifies which part of the object is being uploaded. When a part is uploaded successfully, Amazon S3 returns an entity tag (ETag) for the part as a header in the response. Each part has a part number and a corresponding ETag upon successful upload.

3. **Complete Upload**:

After all parts are uploaded, the final complete multipart upload request is made to the Amazon S3 server. This request includes a list of parts (ETags) that make up the object.

Additionally, you can list the uploaded parts to verify if a part was uploaded successfully to the S3 server. This can help in re-uploading a specific part of the object if needed. Each request returns at most 1,000 multipart uploads.

Downloading objects from S3 Bucket

1. Using getObject() function from aws-sdk package with following params, Bucket,Key,Range.

var params = {

Bucket: 'mybucket',

Key: 'myobject',

Range: 'bytes=0-1024'

};

s3.getObject(params)

References:

- [Amazon S3 Multipart Upload Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html)

- [Multipart Upload JS Demo on GitHub](https://github.com/Tonel/multipart-upload-js-demo)

- [LogRocket Blog on Multipart Uploads with S3, Node.js, and React](https://blog.logrocket.com/multipart-uploads-s3-node-js-react/)

https://stackoverflow.com/questions/26964626/specify-byte-range-via-query-string-in-get-object-s3-request