Go中最快的数据加载方式

问题描述：

I need to regularly load over 300'000 rows x 78 columns of data into my Go program.

Currently I use (import github.com/360EntSecGroup-Skylar/excelize):

xlsx, err := excelize.OpenFile("/media/test snaps.xlsm")
if err != nil {
    fmt.Println(err)
    return
}

//read all rows into df
df := xlsx.GetRows("data")

It takes about 4 minutes on a decent PC using Samsung 960 EVO Series - M.2 Internal SSD.

Is there a faster way to load this data? Currently it takes me more time to read the data than processing it. I'm also opened to other file formats.

我需要定期将300'000行x 78列 code>数据加载到我的转到程序。 p>

当前我正在使用（ import github.com/360EntSecGroup-Skylar/excelize）：


  
   xlsx，错误：= excelize.OpenFile（“ / media / test snaps.xlsm”）
if错误！= nil {
 fmt.Println（err）
返回
} 
  
 //将所有行读入df 
df：= xlsx.GetRows（“ data”）
  code>  pre> 
 
 大约需要 4  em>分钟 在使用Samsung 960 EVO系列-M.2内置SSD的体面的PC上。 p> 
 
 
是否有更快的方式加载此数据？ 目前，读取数据比处理数据需要更多时间。 我还打开了其他文件格式。 p> 
  div>

答

As suggested in the comments, instead of using the XLS format, use a custom, fast data format for reading and writing your table.

In the most basic case, just write the number of columns and rows to a binary file, then write all the data in one go. This will be very fast, I have created a little example here which just writes 300.000 by 40 float32s to a file and reads them back. On my machine this takes about 400ms and 250ms (notice that the file is hot in cache after writing it, may take longer on initial read).

package main

import (
    "encoding/binary"
    "os"

    "github.com/gonutz/tic"
)

func main() {
    const (
        rowCount = 300000
        colCount = 40
    )
    values := make([]float32, rowCount*colCount)
    func() {
        defer tic.Toc()("write")
        f, _ := os.Create("file")
        defer f.Close()
        binary.Write(f, binary.LittleEndian, int64(rowCount))
        binary.Write(f, binary.LittleEndian, int64(colCount))
        check(binary.Write(f, binary.LittleEndian, values))
    }()
    func() {
        defer tic.Toc()("read")
        f, _ := os.Open("file")
        defer f.Close()
        var rows, cols int64
        binary.Read(f, binary.LittleEndian, &rows)
        binary.Read(f, binary.LittleEndian, &cols)
        vals := make([]float32, rows*cols)
        check(binary.Read(f, binary.LittleEndian, vals))
    }()
}

func check(err error) {
    if err != nil {
        panic(err)
    }
}

Go中最快的数据加载方式

相关推荐