如何提高Go中逐行读取大文件的速度

LAST EDIT

This is a "line-by-line" solution to the problem that takes trivial time, it prints the entire matching line.

package main

import (
    "bytes"
    "fmt"
    "io/ioutil"
)

func main() {
    dat, _ := ioutil.ReadFile("./jumble.txt")
    i := bytes.Index(dat, []byte("Iforgotmypassword"))
    if i != -1 {
        var x int
        var y int
        for x = i; x > 0; x-- {
            if dat[x] == byte('
') {
                break
            }
        }
        for y = i; y < len(dat); y++ {
            if dat[y] == byte('
') {
                break
            }
        }
        fmt.Println(string(dat[x : y+1]))
    }
}

real    0m0.421s
user    0m0.068s
sys     0m0.352s

ORIGINAL ANSWER

If you just need to see if the string is in a file, why not use regex?

Note: I kept the data as a byte array instead of converting to string.

package main

import (
    "fmt"
    "io/ioutil"
    "regexp"
)

var regex = regexp.MustCompile(`Ilostmypassword`)

func main() {
    dat, _ := ioutil.ReadFile("./jumble.txt")
    if regex.Match(dat) {
        fmt.Println("Yes")
    }
}

jumble.txt is a 859 MB of jumbled text with newlines included.

Running with time ./code I get:

real    0m0.405s
user    0m0.064s
sys     0m0.340s

To try and answer your comment, I don't think the bottleneck is inherently coming from searching line by line, Golang uses an efficient algorithm for searching strings/runes.

I think the bottleneck comes from the IO reads, when the program reads from the file, it is normally not first in line in the queue of reading, therefore, the program must wait until it can read in order to start actually comparing. Thus, when you are reading in over and over, you are being forced to wait for your turn in IO.

To give you some math, if your buffer size is 64 * 1024 (or 65535 bytes), and your file is 1 GB. Dividing 1 GB / 65535 bytes is 15249 reads needed to check the entire file. Where as in my method, I read the entire file "at once" and check against that constructed array.

Another thing I can think of is just the utter amount of loops needed to move through the file and the time needed for each loop:

Given the following code:

dat, _ := ioutil.ReadFile("./jumble.txt")
sdat := bytes.Split(dat, []byte{'
'})
for _, l := range sdat {
    if bytes.Equal([]byte("Iforgotmypassword"), l) {
        fmt.Println("Yes")
    }
}

I calculated that each loop takes on average 32 nanoseconds, the string Iforgotmypassword was on line 100000000 in my file, thus the execution time for this loop was roughly 32 nanoseconds * 100000000 ~= 3.2 seconds.

You might try using goroutines to process multiple lines in parallel:

lines := make(chan string, numWorkers * 2) // give the channel enough room to put lots of things in so the reader isn't blocked

go func(scanner *bufio.Scanner, out <-chan string) {
  for scanner.Scan() {
    out <- scanner.Text()
  }
  close(out)
}(scanner, lines)

var wg sync.WaitGroup
wg.Add(numWorkers)

for i := 0; i < numWorkers; i++ {
  go func(in chan<- string) {
    defer wg.Done()
    for text := range in {
      if strings.Contains(text, "Iforgotmypassword") {
        fmt.Println(scanner.Text())
      }
    }
  }(lines)
}

wg.Wait()

I'm not sure how much this will really speed things up as it depends on what kind of hardware you have available; it sounds like you're looking for a more than 5x speed improvement, so you might notice if you're running something that can support 8 parallel worker threads. Feel free to use lots of worker-goroutines. Good luck.

Using my own 700MB test file with your original, time was just over 7 seconds

With grep it was 0.49 seconds

With this program (which doesn't print out the line, it just says yes) 0.082 seconds

package main

import (
    "bytes"
    "fmt"
    "io/ioutil"
    "os"
)

func check(e error) {
    if e != nil {
        panic(e)
    }
}
func main() {
    find := []byte(os.Args[1])
    dat, err := ioutil.ReadFile("crackstation-human-only.txt")
    check(err)
    if bytes.Contains(dat, find) {
        fmt.Print("yes")
    }
}