Receive File Upload and Pipe to S3 Bucket Using Presigned Url

In web and mobile applications, information technology's common to provide users with the ability to upload data. Your application may allow users to upload PDFs and documents, or media such every bit photos or videos. Every modern web server engineering science has mechanisms to allow this functionality. Typically, in the server-based surround, the process follows this menstruation:

1. The user uploads the file to the application server.
2. The application server saves the upload to a temporary space for processing.
3. The application transfers the file to a database, file server, or object store for persistent storage.

While the process is simple, information technology tin can have significant side-effects on the performance of the web-server in busier applications. Media uploads are typically large, so transferring these can represent a big share of network I/O and server CPU time. You must also manage the country of the transfer to ensure that the unabridged object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For example, in a spider web awarding that specializes in sending holiday greetings, it may feel most traffic only effectually holidays. If thousands of users try to upload media around the same time, this requires y'all to scale out the application server and ensure that at that place is sufficient network bandwidth available.

By directly uploading these files to Amazon S3, you tin can avoid proxying these requests through your application server. This can significantly reduce network traffic and server CPU usage, and enable your application server to handle other requests during decorated periods. S3 as well is highly available and durable, making it an ideal persistent store for user uploads.

In this weblog mail, I walk through how to implement serverless uploads and show the benefits of this arroyo. This pattern is used in the Happy Path spider web awarding. You lot can download the code from this blog post in this GitHub repo.

Overview of serverless uploading to S3

When you upload direct to an S3 saucepan, yous must first request a signed URL from the Amazon S3 service. Y'all can so upload directly using the signed URL. This is two-stride process for your application front end:

1. Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda function. This gets a signed URL from the S3 saucepan.
2. Directly upload the file from the application to the S3 bucket.

To deploy the S3 uploader example in your AWS account:

1. Navigate to the S3 uploader repo and install the prerequisites listed in the README.doc.
2. In a last window, run:
   git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided

3. At the prompts, enter s3uploader for Stack Name and select your preferred Region. In one case the deployment is complete, note the APIendpoint output.The API endpoint value is the base URL. The upload URL is the API endpoint with /uploads appended. For case: https://ab123345677.execute-api.u.s.a.-west-ii.amazonaws.com/uploads.

Testing the application

I show two ways to test this application. The first is with Postman, which allows you to directly call the API and upload a binary file with the signed URL. The 2nd is with a basic frontend application that demonstrates how to integrate the API.

To exam using Postman:

1. First, copy the API endpoint from the output of the deployment.
2. In the Postman interface, paste the API endpoint into the box labeled Enter request URL.
3. Choose Transport.
4. Subsequently the request is consummate, the Trunk department shows a JSON response. The uploadURL attribute contains the signed URL. Copy this attribute to the clipboard.
5. Select the + icon adjacent to the tabs to create a new asking.
6. Using the dropdown, change the method from GET to PUT. Paste the URL into the Enter request URL box.
7. Cull the Torso tab, and then the binary radio push button.
8. Choose Select file and choose a JPG file to upload.
   Choose Transport. You run across a 200 OK response afterwards the file is uploaded.
9. Navigate to the S3 console, and open the S3 bucket created by the deployment. In the bucket, y'all see the JPG file uploaded via Postman.

To test with the sample frontend application:

1. Re-create index.html from the example's repo to an S3 saucepan.
2. Update the object's permissions to go far publicly readable.
3. In a browser, navigate to the public URL of index.html file.
4. Select Choose file and and so select a JPG file to upload in the file picker. Choose Upload image. When the upload completes, a confirmation bulletin is displayed.
5. Navigate to the S3 console, and open up the S3 bucket created by the deployment. In the bucket, you see the second JPG file you uploaded from the browser.

Agreement the S3 uploading process

When uploading objects to S3 from a web application, you must configure S3 for Cantankerous-Origin Resource Sharing (CORS). CORS rules are defined as an XML document on the bucket. Using AWS SAM, y'all can configure CORS as part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Type: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - Head           AllowedOrigins:             - "*"                  

The preceding policy allows all headers and origins – information technology's recommended that you use a more than restrictive policy for product workloads.

In the first step of the process, the API endpoint invokes the Lambda role to make the signed URL request. The Lambda function contains the post-obit lawmaking:

          const AWS = require('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Principal Lambda entry indicate exports.handler = async (event) => {   return await getUploadURL(event) }  const getUploadURL = async office(result) {   const randomID = parseInt(Math.random() * 10000000)   const Primal = `${randomID}.jpg`    // Become signed URL from S3   const s3Params = {     Bucket: process.env.UploadBucket,     Key,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'image/jpeg'   }   const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)   render JSON.stringify({     uploadURL: uploadURL,     Key   }) }                  

This function determines the name, or key, of the uploaded object, using a random number. The s3Params object defines the accepted content type and also specifies the expiration of the key. In this case, the key is valid for 300 seconds. The signed URL is returned as role of a JSON object including the key for the calling application.

The signed URL contains a security token with permissions to upload this single object to this bucket. To successfully generate this token, the lawmaking calling getSignedUrlPromise must have s3:putObject permissions for the bucket. This Lambda function is granted the S3WritePolicy policy to the bucket by the AWS SAM template.

The uploaded object must match the aforementioned file proper noun and content type as defined in the parameters. An object matching the parameters may exist uploaded multiple times, providing that the upload process starts earlier the token expires. The default expiration is 15 minutes just yous may want to specify shorter expirations depending upon your apply instance.

In one case the frontend application receives the API endpoint response, information technology has the signed URL. The frontend awarding then uses the PUT method to upload binary data direct to the signed URL:

          allow blobData = new Hulk([new Uint8Array(array)], {type: 'image/jpeg'}) const result = await fetch(signedURL, {   method: 'PUT',   body: blobData })                  

At this point, the caller application is interacting directly with the S3 service and not with your API endpoint or Lambda role. S3 returns a 200 HTML status code once the upload is complete.

For applications expecting a large number of user uploads, this provides a simple way to offload a large amount of network traffic to S3, away from your backend infrastructure.

Calculation hallmark to the upload process

The current API endpoint is open, bachelor to whatsoever service on the cyberspace. This means that anyone can upload a JPG file in one case they receive the signed URL. In most product systems, developers want to utilize authentication to control who has access to the API, and who tin upload files to your S3 buckets.

You tin can restrict access to this API past using an authorizer. This sample uses HTTP APIs, which support JWT authorizers. This allows yous to command access to the API via an identity provider, which could exist a service such as Amazon Cognito or Auth0.

The Happy Path application only allows signed-in users to upload files, using Auth0 every bit the identity provider. The sample repo contains a second AWS SAM template, templateWithAuth.yaml, which shows how you can add an authorizer to the API:

                      MyApi:     Type: AWS::Serverless::HttpApi     Properties:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audience:                 - https://auth0-jwt-authorizer             IdentitySource: "$request.header.Say-so"         DefaultAuthorizer: MyAuthorizer                  

Both the issuer and audition attributes are provided by the Auth0 configuration. Past specifying this authorizer as the default authorizer, it is used automatically for all routes using this API. Read part 1 of the Inquire Effectually Me series to acquire more about configuring Auth0 and authorizers with HTTP APIs.

After authentication is added, the calling spider web application provides a JWT token in the headers of the request:

          const response = look axios.get(API_ENDPOINT_URL, {   headers: {     Authorization: `Bearer ${token}`         } })                  

API Gateway evaluates this token before invoking the getUploadURL Lambda function. This ensures that just authenticated users can upload objects to the S3 saucepan.

Modifying ACLs and creating publicly readable objects

In the current implementation, the uploaded object is non publicly accessible. To make an uploaded object publicly readable, you must set its access control list (ACL). In that location are preconfigured ACLs available in S3, including a public-read option, which makes an object readable by anyone on the net. Prepare the appropriate ACL in the params object before calling s3.getSignedUrl:

          const s3Params = {   Bucket: procedure.env.UploadBucket,   Primal,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'image/jpeg',   ACL: 'public-read' }                  

Since the Lambda function must have the advisable bucket permissions to sign the request, you must as well ensure that the function has PutObjectAcl permission. In AWS SAM, yous tin add together the permission to the Lambda function with this policy:

                      - Statement:           - Consequence: Allow             Resource: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Action:               - s3:putObjectAcl                  

Conclusion

Many web and mobile applications permit users to upload data, including big media files like images and videos. In a traditional server-based application, this can create heavy load on the application server, and also use a considerable corporeality of network bandwidth.

By enabling users to upload files to Amazon S3, this serverless pattern moves the network load away from your service. This can brand your awarding much more scalable, and capable of handling spiky traffic.

This blog mail walks through a sample application repo and explains the process for retrieving a signed URL from S3. It explains how to the test the URLs in both Postman and in a web application. Finally, I explain how to add together authentication and make uploaded objects publicly accessible.

To learn more, come across this video walkthrough that shows how to upload directly to S3 from a frontend web application. For more serverless learning resource, visit https://serverlessland.com.

jeffrieshiciandold1991.blogspot.com

Source: https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/

Share this post