Posted in Uncategorized

[question] How to write a large file to response body with low memory usage

   February 26, 2021  1 Comment on [question] How to write a large file to response body with low memory usage

Description

Use response writer to write a large tar file (180M) take up too much memory (1.2~1.7G).
I’m new about golang, and I try to search the io memory solution but I can’t find. I hope somebody can help me :0

How to reproduce

r.GET("/download", func(c *gin.Context) {
                ......
                data := buf.Bytes() //the memory take up 500M
		c.Writer.WriteHeader(http.StatusOK)
		c.Header("Content-Disposition", "attachment; filename=a.tar")
		c.Header("Content-Type", "application/octet-stream")
		c.Header("Content-Length", fmt.Sprintf("%%d", len(data)))
		c.Writer.Write(data) //the memory take up 1.2~1.7G
}

Expectations

Low memory usage

Environment

  • go version:go1.13.8 linux/amd64
  • gin version (or commit ref):v1.6.3
  • operating system:Ubuntu 18.04
Full dicussion on github.com

1 possible answer(s) on “[question] How to write a large file to response body with low memory usage”

  1. I guess you could use an auth middleware. Either globally (for all routes) or for a group of routes.

    But if you really need to add some custom logic before serving file, look how does the StaticFile()works :

    func (group *RouterGroup) StaticFile(relativePath, filepath string) IRoutes {
	if strings.Contains(relativePath, ":") || strings.Contains(relativePath, "*") {
		panic("URL parameters can not be used when serving a static file")
	}
	handler := func(c *Context) {
		c.File(filepath)
	}
	group.GET(relativePath, handler)
	group.HEAD(relativePath, handler)
	return group.returnObj()
}

    It’s basically a handler where you call c.File() in it.

    So I guess, this should work for you :

    r.GET("/download", func(c *gin.Context) {
    c.File("my/path/to/file")
}