Home > Backend Development > Golang > How to Stream Large Files to AWS S3 with Go?

How to Stream Large Files to AWS S3 with Go?

Susan Sarandon
Release: 2024-11-08 04:54:02
Original
1086 people have browsed it

How to Stream Large Files to AWS S3 with Go?

Stream File Upload to AWS S3 Using Go

Challenge:

Stream a large multipart/form-data file directly to AWS S3, minimizing memory and disk footprint.

Solution:

To accomplish this, we'll use the S3 Uploader from the github.com/aws/aws-sdk-go library.

Implementation:

  1. Create a new S3 uploader configured with desired parameters (chunk size, concurrency, etc.).
  2. Open the file to be uploaded.
  3. Use the uploader's Upload function to stream the file to S3.
  4. Handle errors or display upload results.

Example Code:

import (
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

func main() {
    // Create an S3 uploader with custom options
    uploader := s3manager.NewUploader(session.Must(session.NewSession()), func(u *s3manager.Uploader) {
        u.PartSize = 5 * 1024 * 1024 // 5MB part size
        u.Concurrency = 2           // 2 concurrent uploads
    })

    // Open the file for upload
    f, err := os.Open("file.txt")
    if err != nil {
        panic(err)
    }
    defer f.Close()

    // Stream file to S3
    result, err := uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String("my-bucket"),
        Key:    aws.String("file.txt"),
        Body:   f,
    })

    if err != nil {
        panic(err)
    }

    // Display uploaded file information
    fmt.Printf("File uploaded to: %s", result.Location)
}
Copy after login

The above is the detailed content of How to Stream Large Files to AWS S3 with Go?. For more information, please follow other related articles on the PHP Chinese website!

source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Latest Articles by Author
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template