如何每隔10秒从Go中读取大文件的最后几行

如何每隔10秒从Go中读取大文件的最后几行

问题描述:

how can I read the last two lines from a big log file without load it into memory completely?

I need read it every 10 secs(On a Win machine)...and I'm stuck trying to read the last lines..

package main

import (
    "fmt"
    "time"
    "os"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for now := range c {
        readFile(MYFILE)
    }
}

func readFile(fname string){
    file, err:=os.Open(fname)
    if err!=nil{
        panic(err)
    }
    buf:=make([]byte, 32)
    c, err:=file.ReadAt(32, ????)
    fmt.Printf("%s
", c)


}

The log file is something like:

07/25/2013 11:55:42.400, 0.559
07/25/2013 11:55:52.200, 0.477
07/25/2013 11:56:02.000, 0.463
07/25/2013 11:56:11.800, 0.454
07/25/2013 11:56:21.600, 0.424
07/25/2013 11:56:31.400, 0.382
07/25/2013 11:56:41.200, 0.353
07/25/2013 11:56:51.000, 0.384
07/25/2013 11:57:00.800, 0.393
07/25/2013 11:57:10.600, 0.456

Thanks!

我如何从大日志文件中读取最后两行而不将其完全加载到内存中? p>

我需要每10秒钟读取一次(在Win机器上)...并且在尝试读取最后几行时我卡住了。 p>

  包main 
 
import(
“ fmt” 
“ time” 
“ os” 
)
 
const MYFILE =“ logfile.log” 
 
func main(){
c:=时间。 暂时勾选(10 * time.Second)
:=范围c {
 readFile(MYFILE)
} 
} 
 
func readFile(fname string){
 file,err:= os.Open(  fname)
 if err!= nil {
 panic(err)
} 
 buf:= make([[byte,32)
c,err:= file.ReadAt(32,????)\  n fmt.Printf(“%s 
”,c)
 
 
} 
  code>  pre> 
 
 

日志文件类似于: p> \ n

  2013/07/25 11:55:42.400,0.559 
07 / 25/2013 11:55:52.200,0.477 
07 / 25/2013 11:56:02.000,0.463 
07 /  25/2013 11:56:11.800、0.454 
07 / 25/2013 11:56:21.600、0.424 
07 / 25/2013 11:56:31.400、0.382 
07 / 25/2013 11:56:41.200、0.353 \  n07 / 25/2013 11:56:51.000,  0.384 
07 / 25/2013 11:57:00.800,0.393 
07 / 25/2013 11:57:10.600,0.456 
  code>  pre> 
 
 

谢谢! p> div>

You can use file.Seek() or file.ReadAt() to almost the end and then Reading forward. You can only estimate where to start seeking unless you can know that 2 lines = x bytes.

You can get the File length by using the os.Stat(name)

Here is an example based on ReadAt, Stat, and your sample log file:

package main

import (
    "fmt"
    "os"
    "time"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for _ = range c {
        readFile(MYFILE)
    }
}

func readFile(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    buf := make([]byte, 62)
    stat, err := os.Stat(fname)
    start := stat.Size() - 62
    _, err = file.ReadAt(buf, start)
    if err == nil {
        fmt.Printf("%s
", buf)
    }

}

I think a combination of File.Seek(0, 2) and File.Read() should work.

The Seek call gets you to the end of file. You can Seek to a position a bit before the EOF to get last few lines. Then you Read till the EOF and just sleep in your goroutine for 10 seconds; next Read has a chance to get you more data.

You can snatch the idea (and the scan-back logic for initially showing few last lines) from GNU tail's source.

Well, this is only a raw idea and maybe not the best way, you should check and improve it, but seems to work...

I hope that experienced Go users could contribute too..

With Stat you can get the size of the file and from it get the offset for use with ReadAt

func readLastLine(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    fi, err := file.Stat()
    if err != nil {
        fmt.Println(err)
    }

    buf := make([]byte, 32)
    n, err := file.ReadAt(buf, fi.Size()-int64(len(buf)))
    if err != nil {
        fmt.Println(err)
    }
    buf = buf[:n]
    fmt.Printf("%s", buf)

}

Some people will come to this page looking for efficiently reading the last line of a log file (like the tail command line tool).

Here is my version to read the last line of a big file. It use two previous suggestions (using Seek and file Stat).

It read the file backward, byte by byte (no need to set a buffer size) until finding the beginning of a line or the beginning of the file.

func getLastLineWithSeek(filepath string) string {
    fileHandle, err := os.Open(filepath)

    if err != nil {
        panic("Cannot open file")
        os.Exit(1)
    }
    defer fileHandle.Close()

    line := ""
    var cursor int64 = 0
    stat, _ := fileHandle.Stat()
    filesize := stat.Size()
    for { 
        cursor -= 1
        fileHandle.Seek(cursor, io.SeekEnd)

        char := make([]byte, 1)
        fileHandle.Read(char)

        if cursor != -1 && (char[0] == 10 || char[0] == 13) { // stop if we find a line
            break
        }

        line = fmt.Sprintf("%s%s", string(char), line) // there is more efficient way

        if cursor == -filesize { // stop if we are at the begining
            break
        }
    }

    return line
}