如何将HTTP内容流式传输到文件和缓冲区？

问题描述：

I want to copy the data from an http response to a file and a buffer.

However, I can't quite figure this out.

Initially I had this:

func DownloadS3(hash string, cluster Cluster, tok *oauth.Token, offset, length int64, f *os.File) ([]byte, error) {

    // ... other code not shown ...

    resp, err = DoHTTPRequest("GET", s3tok.URL, nil, reqhdr, defaultClientTimeout)
    if err != nil {
        fmt.Println("Error downloading from cluster:", err)
        return nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200 && resp.StatusCode != 206 {
        return nil, err
    }

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)

    // This actually does copy the whole downloaded data to the file as
    // expected.  However, I didn't expect io.CopyBuffer to not use the entire
    // buffer.  It ends up returning a portion of the file.
    _, err = io.CopyBuffer(f, resp.Body, buf)

    return buf, err
}

So what I want actually is something like

_, err = io.CopyBuffer(f, io.TeeReader(resp.Body, buf), nil)

Only, I cant pass buf into TeeReader as it doesn't implement the writer interface. I'm sure there is a proper method but I can't find it as I fumble around looking for an efficient way to do this.

How do I do this without allocating buffer after buffer. I was trying to be efficient. i.e. It seems silly to write the file and read it back.

What I've tried but doesn't work as expected.

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)
    _, err = io.CopyBuffer(f, io.TeeReader(resp.Body,bytes.NewBuffer(buf)), nil)
    return buf, nil

答

Just to make the (other) answer clear and complete: it's not io.TeeReader()'s fault that you were not able to copy the complete body to file and have it as a bytes.Buffer, but it's entirely io.CopyBuffer()'s fault.

io.Copy() is the knight and shining armor, the one who keeps on copying until the whole input is consumed:

Copy copies from src to dst until either EOF is reached on src or an error occurs.

So an equivalently good solution using io.TeeReader():

buf := bytes.NewBuffer(make([]byte, 0, resp.ContentLength))
_, err = io.Copy(f, io.TeeReader(resp.Body, buf))
return buf.Bytes(), err

Using bytes.NewBuffer() like this you can pre-allocate the necessary buffer and avoid reallocation and copying (which makes it faster). Of course you can use this with io.MultiWriter() too.

答

Use io.MultiWriter and bytes.Buffer to grab a copy of the data as the data is written to the file:

var buf bytes.Buffer
_, err := io.Copy(io.MultiWriter(f, &buf), resp.Body)
return buf.Bytes(), err

There is no guarantee that io.CopyBuffer will write to the entire buffer. The current implementation will only use the entire buffer when the entire response body is read in a single call to io.Reader.Read. In your example, it takes more than one read to slurp on the entire response body.

如何将HTTP内容流式传输到文件和缓冲区？

相关推荐