Does Uploading a 10MB File in Go Really Use 10MB of Memory?

The short answer: It depends on your configuration and implementation.

Today I saw an interesting question in the community:

Why does PHP handle file uploads using move_uploaded_file to move a pre-uploaded file, rather than reading the file content directly from the HTTP Body?

This question intrigued me. After researching, I found an analysis of PHP file upload implementation (RFC1867) by Laruence (Bird Brother), but it didn’t fully explain the reason. So I decided to examine Go’s file upload implementation.

Go Implementation

In Go, retrieving uploaded files is straightforward using http.Request.FormFile:

package main

import (
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/files", func(writer http.ResponseWriter, request *http.Request) {
		// 32MB memory limit
		err := request.ParseMultipartForm(32 << 20)
		if err != nil {
			log.Println(err)
			return
		}
		
		// Get uploaded file
		file, handler, err := request.FormFile("file_key")
		log.Println(file, handler, err)
	})
	
	if err := http.ListenAndServe(":8000", nil); err != nil {
		log.Println(err)
	}
}

The implementation of http.Request.FormFile is simple - it retrieves data from a map.
The core logic resides in http.Request.ParseMultipartForm:

func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
	if r.MultipartForm == multipartByReader {
		return nil, nil, errors.New("http: multipart handled by MultipartReader")
	}
	if r.MultipartForm == nil {
		err := r.ParseMultipartForm(defaultMaxMemory)
		if err != nil {
			return nil, nil, err
		}
	}
	if r.MultipartForm != nil && r.MultipartForm.File != nil {
		if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
			f, err := fhs[0].Open()
			return f, fhs[0], err
		}
	}
	return nil, nil, ErrMissingFile
}

The http.Request.ParseMultipartForm method calls multipart.Reader.ReadForm to parse the request body.
Key observations:
- File storage (memory vs disk) depends on whether the total size exceeds the maxMemory parameter.
- Form values count against a separate 10MB allowance within maxMemory.
- If content exceeds maxMemory, files are written to temporary storage (similar to PHP’s approach).

func (r *Reader) readForm(maxMemory int64) (_ *Form, err error) {
	// ... (initialization omitted)

	// Reserve 10MB for non-file parts
	maxValueBytes := maxMemory + int64(10<<20)
	
	for {
		p, err := r.NextPart()
		// ... (error handling omitted)

		if filename == "" {
			// Handle form values
			io.CopyN(&b, p, maxValueBytes+1)
			// ... 
		} else {
			// Handle file upload
			n, err := io.CopyN(&b, p, maxMemory+1)
			
			if n > maxMemory {
				// Write to temp file when exceeding memory limit
				file, err := os.CreateTemp("", "multipart-")
				// ...
				io.Copy(file, io.MultiReader(&b, p))
			} else {
				// Store in memory
				fh.content = b.Bytes()
			}
		}
	}
	return form, nil
}

Key Takeaways

Memory usage depends on your configured maxMemory and actual file sizes
Files exceeding maxMemory get streamed to temporary files
The request body is read sequentially - you can’t skip file processing

Why Can’t We Skip File Processing?

HTTP bodies are parsed sequentially
Files might appear at any position in the multipart data
The server must process each part in order, even for large files

PS: In Go, the Request.Body can only be read once due to its streaming nature.