Skip to main content

Efficient File Upload Solution Based on Go and S3 - Supports Resumable Upload and Instant Upload

In modern App and Web applications, file upload is a common and critical feature. In order to deal with interruptions, repeated uploads, and other problems that may occur during transmission, we designed and implemented a file upload mechanism based on Go Backend + S3 Storage. This solution supports resumable upload and instant upload, and features good scalability and an extremely simple client SDK, enabling direct connection between the client and S3, with the data stream not transitioning through the API service.

✨ Design Goals​

  • ✅ Extremely simple SDK: Clients do not need to adapt to storage system logic.
  • ✅ Reduce server load: File traffic does not go through the API service.
  • ✅ Support resumable upload and instant upload to improve user experience.
  • ✅ Compatible with any S3 interface, easy to access various object storages.
  • ✅ Scalable to support non-S3 HTTP PUT multipart uploads.

📦 Upload Process​

The upload process is divided into five steps:

1. Request Part Size limits​

Interface: GET /object/part_limit

Before the client uploads for the first time, it requests the recommended part size (such as 5MB) from the backend and caches it.

2. Calculate File Hash​

The client splits the file according to the part size, and calculates the hash value of each part. Then it concatenates all part hashes, and finally calculates the overall file hash (file_hash), used for instant upload judgment.

3. Initialize Upload​

Interface: POST /object/initiate_multipart_upload

The client submits the overall file information (such as hash, filename, size):

  • If the server finds that the file already exists (judged by file_hash), it returns the download address indicating successful instant upload.
  • If it does not exist, it returns the upload signature information for each part, including:
    • Part upload URL
    • Required Headers (such as Content-Type, Authorization)
    • Identifiers such as PartNumber

4. Upload Part (Direct Upload to S3)​

Using the returned signature information, the client directly uploads each part to S3 via HTTP PUT:

  • Record the ETag and PartNumber of each part during the upload process.
  • The SDK persists this information to achieve resumable uploads.

⚠️ File content never goes through the backend; only signatures and metadata go through the API.

5. Complete Upload​

Interface: POST /object/complete_multipart_upload

After the client successfully uploads all parts, it calls this interface to submit the list of part information.

  • The backend verifies each part's hash and the overall hash.
  • After passing verification, it starts the S3 merge operation.
  • Finally, it returns the file's access address (which can jump to a signed S3 URL).

⬇️ Download Process​

When the client downloads, it gets a temporary signed download address (S3 Pre-signed URL) from the backend, which can set permissions and valid period control. This address can unify access control verification via an API redirect.

✅ Feature Glance​

FeatureDescription
Instant Upload SupportAchieve skipping upload based on file hash
Resumable UploadSupport resuming upload after disconnection or abnormal exit
Extremely Simple SDKThe client does not need signature logic, only executes PUT
Strong Storage CompatibilitySupports any object storage compatible with HTTP PUT
Highly ScalableCan quickly integrate with other upload protocols and storage systems

⚙️ Scalability Design​

This solution is based on the core architecture of "unifying signature generation on the server + universal PUT upload on the client". As long as the target storage system supports multipart upload and accepts signed PUT requests, it can be seamlessly integrated:

  • Supports Amazon S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS, etc.
  • Can also be extended to support other upload services.

The client code remains unchanged; just extend the signature generation logic on the backend.